---
title: Realtime Adaptive Pooling (RAP)
url: https://www.emergentmind.com/topics/realtime-adaptive-pooling-rap
type: topic
---

# Realtime Adaptive Pooling (RAP)

Searching arXiv for the cited RAP/adaptive pooling papers and related work to ground the article.
Realtime Adaptive Pooling (RAP) denotes a family of runtime pooling strategies that adapt aggregation to the current input in order to improve expressivity, robustness, or streaming suitability. The term is not uniform across the literature. In transformer representation learning, RAP refers to per-token prediction of pooling weights and support size before the next attention layer; in online brain–computer interfaces, it denotes a parameter-free re-parameterization of pooling layers so that offline convolutional decoders can operate on overlapping sliding windows; in several other works, the underlying mechanism is RAP-like even when the paper itself does not use the term [2207.01844], [2507.06779], [2506.09215].

## 1. Terminological scope and definitional variants

RAP is best understood as a research umbrella rather than a single canonical operator. Several papers describe adaptive pooling mechanisms that are causal, input-conditioned, or latency-aware, but they do not all use the same name, and some explicitly use the acronym for something else. This terminological heterogeneity is a recurrent source of confusion [2207.01844], [2506.09215], [2507.06779], [2401.10228], [1804.10070].

| Source | Meaning of RAP | Core mechanism |
|---|---|---|
| "Efficient Representation Learning via Adaptive Context Pooling" [2207.01844] | Realtime Adaptive Pooling in the ContextPool setting | Per-token prediction of pooling weights and support size before attention |
| "Robust Noise Attenuation via Adaptive Pooling of Transformer Outputs" [2506.09215] | RAP is an editorial mapping, not the paper’s own term | Single-query attention pooling with streaming updates |
| "Tailoring deep learning for real-time brain-computer interfaces: From offline models to calibration-free online decoding" [2507.06779] | Explicit "realtime adaptive pooling" | Pooling-layer re-parameterization for online sliding-window CNN decoding |
| "RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything" [2401.10228] | RAP-SAM means Real-Time All-Purpose SAM | Mask-adaptive pooling plus gated query updates |
| "Adaptive pooling operators for weakly labeled sound event detection" [1804.10070] | RAP means regularized auto-pool | L2-regularized learnable auto-pooling parameter |

A plausible implication is that RAP should be read as a class of adaptive aggregation schemes whose defining property is online or runtime adaptation of the pooling operation, rather than as a universally standardized module. The main axes of variation are what is being pooled, how the pooling weights are obtained, and whether adaptation changes weights, support, stride, kernel geometry, or all of these simultaneously.

## 2. Per-token RAP before attention: ContextPool

In ContextPool, RAP is implemented by learning, at run time and per token, both how much to pool from neighboring tokens and how broadly to pool before the next attention layer. Given token features $X \in \mathbb{R}^{n \times d}$ with $X=\{x_1,\dots,x_n\}$, ContextPool produces $Y \in \mathbb{R}^{n \times d}$ of the same sequence length, where each $y_i$ encodes adaptively pooled context around $x_i$ [2207.01844].

The core per-token operator is
$$
y_i=\sum_{j \in \mathcal{N}(i)} \alpha_{ij}\,x_j,
\qquad
\alpha_{ij}=\frac{w_j g^i_j}{\sum_{k \in \mathcal{N}(i)} w_k g^i_k}.
$$
Here $w_j$ is a learned content weight, $g^i_j$ is a soft locality mask, and $\mathcal{N}(i)$ denotes token-wise support. The weights and support are predicted jointly by a lightweight module,
$$
\{w,s\}=m(X),
$$
where $m(\cdot)$ is implemented as two small convolutional layers over the token axis. The support score $s_i \in [0,1]$ is mapped to a Gaussian mask width,
$$
\sigma_i = r\,n\,s_i,
\qquad
g^i \sim \mathcal{N}(i,\sigma_i^2),
$$
with empirically chosen $r=0.1$. In the paper’s default integration, ContextPool is inserted after each attention block and the pooled tokens $Y^\ell$ feed the attention of layer $\ell+1$ rather than being fused back into the current layer by default.

This design changes attention granularity without changing sequence length. Attention still operates over $n$ tokens, but those tokens are now context-enriched summaries whose effective receptive fields vary per position. In transformers, the next attention block receives
$$
Q=YW^q,\quad K=YW^k,\quad V=YW^v,
$$
followed by standard attention,
$$
\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V.
$$
The same principle is extended to ConvNets by replacing fixed pooling with learned weights $W$ and adaptive support $S$ over feature maps, using a 2D Gaussian mask.

The paper emphasizes both expressivity and cost-efficiency. The predictor cost is $O(n \cdot d \cdot c)$ with a small constant. Naively, pooling is $O(n^2 \cdot d)$, but the reported overhead on ViT-B/16 is approximately $1.3$G FLOPs over $55.4$G, or about $2.3\%$, and throughput on V100 changes from $85.9$ to $84.1$ images/s. On WMT14 EN-DE Base NMT, speed changes from $1.20$ to $1.12$ steps/s and memory from $17.2$G to $17.6$G, while BLEU improves from $28.16$ to $28.91$. In performance–cost trade-offs, CP with $L=8$ matches vanilla $L=10$ BLEU while being $27\%$ faster and using $16\%$ less memory. On character-level language modeling, CP-Transformer $(12)$ reaches test BPC $0.99/1.09$ on enwik8/text8, and CP-Transformer $(14)$ reaches $0.97/1.07$. Ablations further report that learned softmax-normalized weights outperform uniform and unnormalized weights, the Gaussian locality prior is essential, adaptive windows outperform fixed windows, CNN-based predictors are the best cost–accuracy choice against MLP or self-attention predictors, and nonlocal pairwise weights are slower and less competitive.

## 3. Query-adaptive pooling of transformer outputs

A distinct RAP formulation appears in "Robust Noise Attenuation via Adaptive Pooling of Transformer Outputs" [2506.09215], where pooling summarizes a transformer output set $X \in \mathbb{R}^{N \times d}$ in the presence of a signal subset $X_s$ and distractor subset $X_\eta$. The paper frames global vector pooling as a differentiable one-cluster vector quantizer,
$$
C(X)=\sum_{i=1}^N w_i \odot x_i,
$$
and defines signal loss as
$$
\mathcal{L}(X,x_c)=\frac{1}{k}\sum_{x_s \in X_s}(x_s-x_c)^2.
$$
Under this objective, the signal-optimal compressor outputs the centroid of the signal subset, with optimal weights $w_i^\ast = 1/k$ for signal tokens and $0$ for noise tokens.

AdaPool realizes this objective through single-query cross-attention. With query $x_q$,
$$
r(x_q,x_i)=\langle \phi_\theta(x_q),\psi_\theta(x_i)\rangle
= x_q W_Q W_K^\top x_i^\top \cdot \frac{1}{\sqrt{d}},
$$
$$
w_i=\frac{\exp(r_i)}{\sum_{j=1}^N \exp(r_j)},
\qquad
\mathrm{AdaPool}(X)=\sum_{i=1}^N w_i \cdot x_i W_V.
$$
The paper states that AvgPool and MaxPool are special cases of AdaPool, and that ClsToken differs from AdaPool only by the use of a fixed learned query. Section 3.5 argues that the pooling query should be sourced from the signal subset $X_s$ so that signal tokens receive larger relation scores than noise tokens. Theorem 3.1 then bounds the deviation between learned weights and signal-optimal weights in terms of the intra-signal spread $\epsilon_s$, intra-noise spread $\epsilon_\eta$, and the margin $M=\min\{r_s\}-\max\{r_\eta\}$.

The real-time mapping arises from an online normalization identity. For fixed query $x_q$, define
$$
u=\frac{x_q W_Q W_K^\top}{\sqrt{d}},
\qquad
r_i = u x_i^\top.
$$
Maintaining
$$
S_t=\sum_{j=1}^t \exp(r_j), \qquad
Y_t=\sum_{j=1}^t \exp(r_j)(x_j W_V),
$$
gives the exact pooled output
$$
y_t=\frac{Y_t}{S_t}.
$$
With an arriving token $x_t$,
$$
s_t=\exp(r_t), \quad S_t=S_{t-1}+s_t,\quad Y_t=Y_{t-1}+s_t(x_t W_V),
$$
$$
y_t=\left(\frac{S_{t-1}}{S_t}\right)y_{t-1}+\left(\frac{s_t}{S_t}\right)(x_t W_V).
$$
This produces a causal single-pass RAP operator with memory cost $O(d)$ for $y_t$ and $O(1)$ for $S_t$, and no quadratic $n^2$ attention over the pooled sequence.

The empirical results are organized around robustness to changing signal-to-noise ratio. On the synthetic KNN-Centroid task with $1$M samples, $N=128$, and $d=16$, AdaPool achieves the lowest and most stable MSE across SNRs. At KNN-4, AdaPool records $0.006 \pm 0.001$ against AvgPool $0.055$, MaxPool $0.050$, and ClsToken $0.063$; at KNN-32, it records $0.002$ against $0.016$, $0.055$, and $0.027$; at KNN-128, AdaPool and AvgPool both reach $0.000$ while MaxPool remains at $0.046$. In multi-agent RL, relational reasoning, and CIFAR-10/100, the paper reports that adaptive pooling remains the most robust regime as distractor counts increase, while query choice materially affects performance: on CIFAR-100, Ada-Mean reaches $61.23 \pm 0.20$, Ada-Focal $61.22 \pm 0.33$, MaxPool $60.55 \pm 0.28$, AvgPool $59.63 \pm 0.23$, ClsToken $55.56 \pm 0.13$, and Ada-Corner $57.08 \pm 0.31$.

## 4. RAP as pooling re-parameterization for online BCI decoding

In online BCI decoding, RAP has a different meaning. It is a parameter-free method that modifies the pooling layers of offline-trained CNNs so that the same network can decode overlapping sliding windows in real time, without changing learned convolution, classifier, or normalization weights [2507.06779].

The formulation treats pooling as an operator on an intermediate temporal feature map. For an input segment $X \in \mathbb{R}^{C \times (T f_s)}$, the first $P-1$ pooling layers perform downsampling with kernel size equal to stride,
$$
f_{\mathrm{inter}} = \frac{f_s}{\prod_{i=1}^{P-1} k_i},
\qquad
k_i=s_i \ \text{for}\ i \le P-1.
$$
The final pooling layer becomes a window-extraction stage aligned with the online update rate $f_u$ and window length $T_w$:
$$
k_P = f_{\mathrm{inter}} \cdot T_w,
\qquad
s_P = \frac{f_{\mathrm{inter}}}{f_u}.
$$
Given an intermediate feature map $Z \in \mathbb{R}^{F \times (T f_{\mathrm{inter}})}$, the last pooling layer outputs
$$
Y_t=P(Z_{t-k_P+1:t}),
$$
for
$$
t=k_P,\ k_P+s_P,\ \dots,\ k_P+(N_w-1)s_P,
$$
with
$$
N_w=(T-T_w)f_u+1.
$$

The central claim is equivalence between joint training-time decoding of all windows in a trial and single-window inference at test time, up to negligible padding differences. This lets a model process a full trial once during training, while still satisfying low-latency online inference constraints at deployment. The temporal compute reduction factor over naïve sliding-window training is
$$
\mathrm{Reduction}=\frac{N_w T_w f_s}{T_t f_s}
= \left(\frac{T_w}{T_t}\right) N_w.
$$
For Dreyer2023, with $T_t=4.75$ s, $T_w=1$ s, $f_u=16$ Hz, and $N_w=61$, the reduction is approximately $12.84\times$; for Lee2019, with $T_t=3.75$ s and $N_w=45$, it is $12\times$.

The reported instantiation uses BaseNet with $P=2$, $f_s=256$ Hz, $f_u=16$ Hz, and $T_w=1$ s. Choosing $k_1=s_1=8$ yields $f_{\mathrm{inter}}=32$ Hz, and the final pooling layer uses $k_2=32$ and $s_2=2$. Inference remains standard single-window forward propagation, one update every $1/f_u$ seconds. On an Intel i7-1195G7 CPU with four cores, inference time per window is reported as $2.15$ ms for BaseNet and $0.5$ ms for RiemannMDM, against a real-time budget of $62.5$ ms at $f_u=16$ Hz.

RAP is combined with source-free domain adaptation rather than replacing it. The paper describes Euclidean alignment and Riemannian alignment using target-specific reference covariances, and Adaptive Batch Normalization with online update
$$
\mu_i=(1-\alpha)\mu_{i-1}+\alpha E[X], \qquad \mu_0=\mu_S,
$$
$$
\sigma_i^2=(1-\alpha)\sigma_{i-1}^2+\alpha \mathrm{Var}[X], \qquad \sigma_0^2=\sigma_S^2,
$$
with $\alpha=0.001$ in experiments. In cross-subject evaluation, the zero-shot source model yields BaseNet trial-wise accuracy $68.3\% \pm 14.3$ on Dreyer2023 and $72.4\% \pm 14.7$ on Lee2019. In online test-time adaptation on Lee2019, BaseNet+RA reaches approximately $76.9\% \pm 14.1$ against $71.4\% \pm 12.5$ for RiemannMDM+GR. The paper also reports stable window-wise accuracy profiles over time and physiologically plausible contralateral C3/C4 discriminancy maps.

## 5. Domain-specific RAP-like operators in time series, vision, audio, video, and point clouds

Several adjacent literatures instantiate the same adaptive-pooling principle in task-specific forms. In "Attention Mamba: Time Series Modeling with Adaptive Pooling Acceleration and Receptive Field Enhancements" [2504.02013], the Adaptive Pooling block fuses adaptive average pooling and adaptive max pooling on $Q$ and $K$, downsamples both last two dimensions to one quarter, applies GeLU, computes pooled similarities, and projects the result back to full resolution. The pooled tensors satisfy $FQ,FK \in \mathbb{R}^{B \times E/4 \times E/4}$, the scores are $\mathrm{Scores}=\mathrm{PoolQ} @ \mathrm{PoolK}$, and the block is paired with a bidirectional Mamba value path. On PEMS07, the model reports $69.09$ ms/iter, $1.560$ GB, and MSE $0.057$, compared with S-Mamba at $98.51$ ms/iter, $1.030$ GB, and MSE $0.063$, and iTransformer at $197.86$ ms/iter, $1.670$ GB, and MSE $0.067$. The paper attributes part of the gain to the enlarged receptive field created by global pooled re-projection.

In video action recognition, AdaScan performs a single temporal scan of the video and continuously predicts a discriminative importance $\gamma_{t+1} \in [0,1]$ for the next frame using a three-layer MLP with tanh nonlinearities and a sigmoid output [1611.08240]. The pooled representation is updated as a normalized weighted mean,
$$
\psi(X,t+1)=\frac{\hat{\gamma}_t \psi(X,t)+\gamma_{t+1}\phi(x_{t+1})}{\hat{\gamma}_{t+1}},
\qquad
\hat{\gamma}_p=\sum_{k=1}^p \gamma_k.
$$
This is causal because $\gamma_{t+1}$ depends only on the current pooled state and the next frame feature. On UCF101 split 1, AdaScan improves spatial-stream accuracy from mean pooling $78.0$ to $79.1$ and temporal-stream accuracy from $80.8$ to $81.7$; the two-stream model reaches $89.4$ on UCF101 and $54.9$ on HMDB51, increasing to $91.3/61.0$ with iDT late fusion and $93.2/66.9$ with iDT + C3D late fusion.

In segmentation, the mechanism in RAP-SAM is not called Realtime Adaptive Pooling; the paper states that RAP-SAM means Real-Time All-Purpose SAM [2401.10228]. Its decoder repeatedly predicts masks, pools encoder features with those masks,
$$
X_i=\sum_{u,v} M_{i-1}(u,v)\,F_{\mathrm{img}}(u,v)
$$
for images and
$$
X_i=\sum_{t,u,v} M_{i-1}(t,u,v)\,F_{\mathrm{vid}}(t,u,v)
$$
for videos, then updates queries through gated blending and MHSA. This replaces heavy per-pixel cross-attention with mask-adaptive pooling. At COCO-scale input $1333 \times 800$, RAP-SAM with ResNet-50 reports PQ $46.9$, SQ $80.8$, PQ$_{\mathrm{th}}$ $51.6$, PQ$_{\mathrm{st}}$ $39.8$, interactive mIoU $57.9$, YT-VIS mAP $46.2$, $123$ GFLOPs, $47.2$M parameters, and $35.1$ FPS; the ResNet-18 version reports $40.3$ FPS.

In weakly labeled sound event detection, the paper’s own RAP acronym means regularized auto-pool, not realtime adaptive pooling [1804.10070]. Auto-pool learns a scalar $\alpha$ that interpolates between mean-, max-, and min-like aggregation:
$$
w_t=\frac{\exp(\alpha f_t)}{\sum_j \exp(\alpha f_j)},
\qquad
y=\sum_t w_t f_t.
$$
Regularized auto-pool adds $\lambda\|\alpha\|_2^2$. The paper reports that auto-pool consistently achieves the best static clip-level F1, CAP reaches dynamic F1 $0.533$ against strong-label $0.551$ on URBAN-SED, and RAP with $\lambda=10^{-3}$ reaches dynamic F1 approximately $0.673$ against strong-label $0.675$ on MedleyDB. Although this is a MIL setting rather than a streaming decoder, it formalizes adaptive pooling as a class-dependent control of pooling sharpness.

Point-cloud work replaces max or mean aggregation with mode-seeking pooling over each feature channel [2106.10850]. The histogram-based variant detects the densest bin and pools over its inliers. In the reported tensor setting $10 \times 1024 \times 2048$, histogram pooling uses $2$ GB, test time $3$ s, and train time $9$ s/epoch, compared with max pooling at $2$ GB, $3$ s, and $11$ s/epoch. Under $50\%$ uniform outliers on ModelNet40 test data, the histogram method reaches approximately $70\%$ accuracy. The paper explicitly presents the histogram implementation as the real-time variant.

Learnable pooling can also be implemented as a small recurrent operator. In "A Fully Trainable Network with RNN-based Pooling" [1706.05157], a single-neuron LSTM scans an $N \times N$ window and outputs the final hidden state as the pooled value. The paper reports mean absolute errors on synthetic max-pooling regression of approximately $8.9 \times 10^{-5}$ for $2 \times 2$ windows, $4.4 \times 10^{-5}$ for $3 \times 3$, and $5.3 \times 10^{-5}$ for $4 \times 4$. On CIFAR-10, FTN-Layer reduces test error from $32.75$ to $25.72$ in Conv$_{16}$ and from $57.44$ to $50.52$ in Conv$_8$. On a Titan X GPU, test time per sample for a Conv$_{64}$-like network is $0.0068$ s for FTN against $0.0027$ s for the baseline CNN.

## 6. Limitations, misconceptions, and research directions

A first misconception is that RAP denotes one specific algorithm. The literature instead contains several incompatible definitions: per-token support-adaptive pooling before attention, single-query cross-attention pooling over transformer outputs, parameter-free stride/kernel reconfiguration for online CNN decoding, mask-adaptive ROI-like pooling, and regularized auto-pooling in MIL [2207.01844], [2506.09215], [2507.06779], [2401.10228], [1804.10070]. This suggests that RAP is more accurately treated as a design pattern centered on runtime adaptation of aggregation.

A second misconception is that adaptivity automatically implies negligible overhead. In ContextPool, predictor and pooling overheads are small but nonzero, and naïve pooling remains $O(n^2)$ in long sequences unless support is bounded or sparsified [2207.01844]. In AdaPool, the online update is causal and exact, but theoretical guarantees depend on positive signal–noise margin $M$ and bounded intra-class spreads $\epsilon_s,\epsilon_\eta$; when signal and noise overlap, the guarantees degrade [2506.09215]. In the BCI formulation, RAP does not make convolutions incremental: the paper explicitly notes that if true streaming convolution without full-window recomputation is required, additional caching and incremental convolution updates are needed [2507.06779].

A third issue is that adaptive pooling can fail for domain-specific reasons. Attention Mamba fixes the downscale ratio to one quarter, and the paper notes reduced effectiveness when the embedding dimension is small or mismatched to the number of variates; it also reports a $34\%$ memory increase over S-Mamba on PEMS07 [2504.02013]. RAP-SAM’s pooling can over-smooth boundaries and degrade in crowded or overlapping scenes, which motivates its prompt-specific cross-attention adapter [2401.10228]. ContextPool reports that tasks dominated by strictly local dependencies may benefit less, recommending smaller $r$ or applying RAP only in deeper layers [2207.01844].

The dominant research directions are correspondingly heterogeneous. Several papers explicitly propose bounded-support or sparse variants for latency control, multi-head or subspace-specific pooling, uncertainty-aware weighting, and online tuning of softmax sharpness or gating thresholds [2506.09215], [2207.01844]. The BCI formulation points to broader applicability in time-series CNNs for speech, wearable sensing, and industrial monitoring, provided that sampling rate $f_s$, update frequency $f_u$, and window length $T_w$ admit a valid pooling re-parameterization [2507.06779]. A plausible implication is that future RAP research will likely continue to split along two lines: adaptive pooling as a representational primitive inside deep sequence models, and adaptive pooling as a systems technique for reconciling offline architectures with online deployment constraints.

Source: https://www.emergentmind.com/topics/realtime-adaptive-pooling-rap