---
title: 'On-fly Pooling Module: Adaptive Aggregation'
url: https://www.emergentmind.com/topics/on-fly-pooling-module
type: topic
---

# On-fly Pooling Module: Adaptive Aggregation

Searching arXiv for recent papers on strip pooling and related “on-the-fly” pooling modules.
arxiv_search(query="strip pooling scene parsing on-the-fly pooling module", max_results=5)
arxiv_search(query="stochastic average pooling plug-and-play pooling module", max_results=5)
In the literature represented by recent arXiv work, an **On-fly Pooling Module** is best understood not as a single canonical operator but as a family of **plug-and-play pooling or pooling-attention components** computed during the forward pass and attached to existing backbones without structural overhaul. This usage includes directional strip pooling for scene parsing, multi-kernel pooling for retinal vessel segmentation, stochastic pooling-as-regularization, learnable nonuniform pooling for gaze estimation, non-local self-attentive down-sampling, edge-focused Max–Min pooling, and distribution-aware graph readout [2003.13328], [2204.03213], [2409.16630], [1903.05761], [2209.07659], [2109.14333]. This suggests that “on-fly pooling” is primarily a **design principle**: pooling is made task-aware, input-conditioned, or reconfigurable at runtime, rather than remaining a fixed \(N\times N\) reduction.

## 1. Scope and terminology

A recurrent property of these modules is **drop-in compatibility**. The strip pooling paper explicitly states that the Strip Pooling Module (SPM) “can be directly applied to any pretrained backbone networks *without training them from scratch*,” while preserving the input-output tensor shape \(C\times H\times W \rightarrow C\times H\times W\) [2003.13328]. The same plug-and-play framing appears in DKEPool for graph readout, OPFR for point clouds, and stochastic average pooling as a replacement for existing average pooling layers [2109.14333], [2407.21335], [2409.16630].

A common misunderstanding is to equate *on-fly* with **explicit dynamic architecture search or data-dependent kernel selection**. Several modules surveyed here are **static in architecture** even when they are computed on each forward pass. MC-UNet’s multi-kernel pooling uses a fixed kernel set \(\{2,3,5,6\}\), and strip pooling fixes horizontal and vertical strip orientations; the per-input variation comes from feature values or attention masks rather than from changing the branch topology itself [2204.03213], [2003.13328]. By contrast, LPM predicts nonuniform pooling borders from the input image, and SAP randomizes the subset of elements used during training, so these are on-fly in a stronger sense [1903.05761], [2409.16630].

The term therefore spans at least three regimes. One regime uses **runtime-computed but architecturally fixed** pooling, as in strip pooling and multi-kernel pooling. A second regime uses **stochastic or adaptive pooling layouts**, as in SAP and LPM. A third regime uses pooling as a **learned aggregation mechanism over structured sets**, such as graph nodes or point-cloud neighborhoods [2003.13328], [2409.16630], [2109.14333], [2407.21335].

## 2. Directional and multi-scale spatial pooling

The most explicit formulation of an on-fly spatial pooling block appears in "Strip Pooling: Rethinking Spatial Pooling for Scene Parsing" [2003.13328]. Conventional spatial pooling uses isotropic square kernels \(N\times N\), whereas strip pooling uses **long but narrow kernels**, \(1\times N\) or \(N\times 1\). For a 2D feature map \(\mathbf{x}\in\mathbb{R}^{H\times W}\), horizontal and vertical strip pooling are defined as
\[
y_i^h = \frac{1}{W}\sum_{0 \le j < W} x_{i,j}, \qquad
y_j^v = \frac{1}{H}\sum_{0 \le i < H} x_{i,j}.
\]
This gives **1D global context per row and per column** while preserving locality in the orthogonal dimension. In SPM, these row and column priors are refined by 1D convolutions of kernel size 3, fused as
\[
y_{c,i,j}=y^h_{c,i}+y^v_{c,j},
\]
and converted into an attention-like mask through a \(1\times1\) convolution and sigmoid:
\[
\mathbf{z}=\text{Scale}\big(\mathbf{x},\, \sigma(f(\mathbf{y}))\big).
\]
The module is inserted after the \(3\times3\) convolution of the last residual block in each ResNet stage and after all residual blocks in the last stage, so long-range context is added as a residual enhancement rather than by redesigning the backbone [2003.13328].

The same paper introduces the **Mixed Pooling Module (MPM)**, a residual bottleneck that combines long-range dependency aggregation via strip pooling with short-range dependency aggregation via lightweight pyramid pooling [2003.13328]. Its short-range branch uses three paths: original resolution with a \(3\times3\) convolution, pooling to \(20\times20\) bins plus \(3\times3\) convolution, and pooling to \(12\times12\) bins plus \(3\times3\) convolution. The long-range branch applies strip pooling after a \(1\times1\) channel reduction. This design makes the module repeatable inside the feature extractor, unlike PSP-style heads.

A related but distinct multi-scale formulation appears in MC-UNet, where the **Multi-Kernel Pooling (MKP)** module is “based on the spatial pyramid” and applies parallel **max pooling** with kernel sizes \(2\times2\), \(3\times3\), \(5\times5\), and \(6\times6\), followed by \(1\times1\) convolution, upsampling, and fusion with the original feature map [2204.03213]. The paper presents
\[
\text{MKP}=\sum f_1(\text{Maxpool}_{k_i}(D)),
\]
while the text states that the original features and upsampled pooled features are concatenated. This suggests that MKP functions as a static but forward-pass-computed context pyramid.

Earlier work on **stacked pooling** and **multiple max-pooling integration** addresses the same multi-scale objective from different directions. "Stacked Pooling: Improving Crowd Counting by Boosting Scale Invariance" defines multi-kernel pooling as the element-wise mean of pooled outputs with multiple receptive fields and shows that stacked pooling is an equivalent but cheaper sequential realization [1808.07456]. MC-Net’s multiple max-pooling integration module, by contrast, pools encoder features from several depths to a common spatial size and concatenates them for the decoder, with a \(1\times1\) convolution used to integrate channels before max pooling of different sizes [2003.11213]. In both cases, pooling is not merely down-sampling; it is a mechanism for multi-scale feature integration.

Adaptive Context Encoding (ACE) generalizes this line of work by replacing fixed pooling sizes and fixed atrous rates with **deformable convolution**, so the effective aggregation field becomes input dependent [1907.06082]. ACE uses three deformable convolution blocks, each followed by BatchNorm and ReLU, and is positioned as a replacement for ASPP or PPM. This suggests an important boundary case: an on-fly pooling module need not be a literal pooling operator if it serves the same role of **adaptive context aggregation**.

## 3. Stochastic, learnable, and attention-based pooling

"Stochastic Subsampling With Average Pooling" formalizes an on-fly pooling layer that is randomized during training but deterministic at test time [2409.16630]. For pooling size \(r\) and keep probability \(p\),
\[
\text{SAP}_{\text{test}}^r(x)=\text{AP}^r(x), \qquad
\text{SAP}_{\text{train}}^r(x)=\sqrt{p}\,\text{AP}^{rp}\!\bigl(\text{SS}_{\text{train}}(x)\bigr).
\]
The central claim is statistical consistency: stochastic subsampling preserves mean and variance for large vectors, and the \(\sqrt{p}\) factor matches the second moment of train-time and test-time pooling. The module is intended as a drop-in replacement for average pooling in classifier heads, SE-style attention blocks, pyramid pooling, and detection necks [2409.16630].

A different notion of on-fly behavior appears in the **Learnable Pooling Module (LPM)** for gaze estimation [1903.05761]. LPM predicts **nonuniform pooling borders** from the input image via a small convolutional layer and a fully connected layer, then averages each learned cell:
\[
y(i,j)=\frac{1}{n_{ij}}\sum_{(u,v)\in \text{cell }(i,j)} x(u,v).
\]
The grid is constrained so borders do not cross, and gradients for border positions are estimated numerically:
\[
\frac{\partial y(i,j)}{\partial p_j} \approx \frac{y'(i,j)-y(i,j)}{h}.
\]
This is an explicitly adaptive pooling layout: valuable regions remain at higher resolution, while less informative regions are pooled more aggressively [1903.05761].

"Self-Attentive Pooling for Efficient Deep Learning" makes the pooling weights themselves **non-local and learned** [2209.07659]. The layer first embeds patches with a strided convolution, applies multi-head self-attention on patch tokens, restores the result to the original spatial-channel resolution, and then performs weighted average pooling:
\[
O=\frac{\sum_{i=p}^{p+s-1}\sum_{j=q}^{q+s-1}\pi_{i,j}(x)\,x_{i,j}}
{\sum_{i=p}^{p+s-1}\sum_{j=q}^{q+s-1}\pi_{i,j}(x)}.
\]
Here, \(\pi(x)\) is obtained from restored self-attention features through sigmoid followed by exponential. The stated objective is aggressive early down-sampling with smaller memory footprints while preserving non-local dependencies [2209.07659].

An edge-specific variant appears in the **Edge Attention Module (EAM)**, whose core operator is **Max–Min pooling**:
\[
(F_{mn}I) = \max_{\text{window}} I - \min_{\text{window}} I.
\]
With a \(5\times5\) window and stride 2, this cancels the local base level and retains only the total deviation within the window, which the paper interprets as edge information [2502.03103]. EAM applies Max–Min pooling to a late feature map, follows it with convolutional refinement and global average pooling, and concatenates the resulting edge descriptor with the backbone descriptor. This is not an attention map in the CBAM sense; rather, it is a parallel edge-focused pooled representation.

The UAV vehicle Re-ID literature offers a closely related attention design. The Dual-pooling Attention module combines Average Pooling, Generalized Mean Pooling, Minimum Pooling, and Soft Pooling in both channel and spatial branches, then refines the result with OD convolution and residual fusion [2306.14104]. This suggests that on-fly pooling can also be framed as **multi-operator attention construction**, where different pooling statistics are fused before weighting the backbone representation.

## 4. Pooling beyond regular image grids

The idea extends naturally to **point clouds**. In "On-the-fly Point Feature Representation for Point Clouds Analysis," OPFR constructs local triangle sets through Hierarchical Sampling, builds approximate local reference frames with LRCon, computes location, orientation, and curvature features with CFGen, and then aggregates them by a shared MLP and a symmetric pooling operator \(\mathcal{A}\) [2407.21335]:
\[
\mathbf{r}_i=
\mathcal{A}\big(\{\mathcal{F}([\mathbf{x}'_{ij};\mathbf{n}_{ij};\mathbf{p}_{ij}]): j=1,\dots,K\}\big).
\]
Ablation shows that **sum pooling performs best**, and the module adds only \(0.012\)M parameters and \(1.56\) ms per sample relative to the PointNet++ baseline [2407.21335]. Here, the pooled object is not a spatial patch but a geometric neighborhood represented through PFH-inspired descriptors.

Graph learning offers a different generalization. DKEPool treats node embeddings as samples from a Gaussian distribution with mean \(\boldsymbol{\mu}\) and covariance \(\boldsymbol{\Sigma}\), and defines the pooled graph representation as
\[
\mathbf{z}=\mathbf{W}^\top \boldsymbol{\Sigma}\boldsymbol{\mu}.
\]
This replaces flat sum or mean readout with a **distribution knowledge embedding** that encodes both the first-order location and second-order dispersion of node features [2109.14333]. Robust DKEPool further applies iterative matrix square-root normalization to covariance before projection. The resulting module is explicitly described as plug-and-play and permutation invariant.

These non-Euclidean cases clarify that on-fly pooling is not restricted to image down-sampling. The common operation is **runtime summarization of a variable-size local or global set**—neighbor points, graph nodes, or feature-map cells—into a compact descriptor that can be appended to any backbone [2407.21335], [2109.14333].

## 5. Reported empirical behavior

The reported empirical pattern is that these modules typically improve performance **without replacing the backbone wholesale**. In strip pooling, the gains arise from directional long-range context at low overhead; in SAP, from regularization without train-test inconsistency; in OPFR, from explicit geometry; and in self-attentive pooling, from non-local weighting during down-sampling [2003.13328], [2409.16630], [2407.21335], [2209.07659].

Representative results reported in the corresponding papers are summarized below.

| Module | Setting | Reported result |
|---|---|---|
| SPNet | ADE20K val, ResNet-50, multi-scale + flip | mIoU 45.03, Pixel Acc 81.32 |
| SPNet | Cityscapes test, ResNet-101, fine annotations only | 82.0% mIoU |
| SPNet | Pascal Context test, ResNet-101 | 54.5% mIoU |
| Base FCN + 2 MPM + SPMs | ADE20K ablation | mIoU 44.03, Pixel Acc 80.65% |
| MC-UNet | DRIVE | ACC 96.78, SEN 81.00, SP 98.79, AUC 98.28, F1 81.49 |
| MC-UNet | CHASE_DB1 | ACC 97.14, SEN 83.66, SP 98.29, AUC 98.18, F1 77.41 |
| PointNet++ + OPFR | ModelNet40 | OA 94.5%, mAcc 91.6% |
| PointNet++ + OPFR | S3DIS Area-5 | mIoU 69.1%, mAcc 76.9%, OA 90.0% |
| Point Transformer + OPFR | S3DIS Area-5 | mIoU 72.6%, mAcc 78.6%, OA 91.7% |
| Self-attentive pooling | MobileNet-V2 on ImageNet | average 1.2% higher test accuracy than existing pooling techniques |
| Self-attentive pooling | iso-memory setting | 1.43% higher test accuracy and up to 22x reduction in memory consumption |
| EAM + CNN | Caltech-101 | accuracy 95.5% |
| EAM + CNN | Caltech-256 | accuracy 86% |

Several ablations directly isolate the pooling mechanism. In strip pooling, replacing strip pooling inside SPM with global average pooling drops the ADE20K score from \(44.03\) mIoU to \(41.34\), below the \(41.92\) baseline with two MPMs, which the paper interprets as evidence that **directional global context** is more effective than collapsing the whole map into a single global vector [2003.13328]. In MC-UNet, the MKP branch improves AUC while the DAC branch improves specificity, and the combined model is strongest overall [2204.03213]. In EAM, replacing Max–Min pooling with standard Max pooling removes much of the gain, especially for DenseNet-121, indicating that the crucial factor is not merely adding a side branch but specifically adding an **edge-selective pooling operator** [2502.03103].

A second empirical pattern is that greater flexibility does not always imply a heavier module. Strip pooling is described as essentially averaging over one dimension plus 1D convolutions, and DKEPool can be used without an explicit projection matrix, while SAP adds no extra parameters at all [2003.13328], [2109.14333], [2409.16630]. The most substantial compute overheads arise when the pooling mechanism includes self-attention, deformable convolution, or numerically delicate covariance normalization [2209.07659], [1907.06082], [2109.14333].

## 6. Limits, misconceptions, and research directions

A persistent misconception is that all on-fly pooling modules are **content-adaptive in the same way**. The surveyed work shows otherwise. Strip pooling and MC-UNet use fixed branch structures and fixed strip or pooling scales; SAP randomizes sampling but keeps deterministic test-time pooling; LPM learns the pooling grid itself; ACE changes the effective context field through deformable offsets; DKEPool changes the object of pooling from vectors to distributions [2003.13328], [2204.03213], [2409.16630], [1903.05761], [1907.06082], [2109.14333]. This suggests that the field is unified more by **runtime aggregation philosophy** than by a single mechanism.

The limitations are similarly heterogeneous. Strip pooling has a **directional bias**, since only horizontal and vertical strips are modeled, and the paper notes diminishing returns when too many SPMs or MPMs are added [2003.13328]. MC-UNet’s MKP remains **multi-scale but static**, with no learned scale routing [2204.03213]. EAM’s Max–Min pooling can respond to noise and may suppress homogeneous foreground texture [2502.03103]. LPM requires numerical gradient estimation for pooling borders and sensitive learning-rate control [1903.05761]. DKEPool relies on a **Gaussian assumption** and incurs covariance-related cost, particularly in its robust variant [2109.14333]. Self-attentive pooling adds non-local modeling power but necessarily introduces more machinery than ordinary average or max pooling [2209.07659].

The future directions named or implied across these papers are consistent. One direction is **adaptive geometry**: shorter or variable-length strips, diagonal strips, deformable strip shapes, or learned pooling regions following object boundaries [2003.13328]. Another is **content-aware branch weighting**, such as scale attention over MKP branches or mixtures of stochastic, average, max, and contrast-style pooling [2204.03213], [2409.16630], [2306.14104]. A third is the extension of specialized pooling ideas into new backbones and modalities, such as injecting edge-focused or problem-specific modules into Vision Transformers, or using explicit geometric pooling with MLP- and Transformer-based point-cloud models [2502.03103], [2407.21335].

Taken together, these studies show that an On-fly Pooling Module is not a single layer type but a broad research program: **replace fixed aggregation with runtime-computed, task-structured pooling** while preserving the engineering advantages of plug-and-play modules. The central trade-off is no longer only between max and average pooling, but between isotropic and directional context, static and adaptive scales, deterministic and stochastic aggregation, local and non-local weighting, and first-order versus distributional summaries [2003.13328], [2409.16630], [2109.14333].

Source: https://www.emergentmind.com/topics/on-fly-pooling-module