Sequence Pooling: Methods & Applications

• Sequence pooling is a family of techniques that reduces variable-length feature sequences into fixed-size embeddings by aggregating both local and global information.
• Approaches range from fixed operators like [CLS] token and mean pooling to advanced methods such as landmark, attention-based, and learned pooling to address bias and information dilution.
• Empirical evaluations show that methods like LMK pooling enhance performance in retrieval and classification tasks, especially in handling long contexts with minimal computational overhead.

Sequence pooling encompasses a family of operators that reduce a variable-length sequence of feature vectors or hidden states to a compact, fixed-size representation. This process underpins dense retrieval, classification, detection, and sequence modeling across domains, providing a mechanism for invariant summarization, dimensionality reduction, and efficient computation in both shallow and deep models. The choice and design of sequence pooling are critical, as conventional approaches such as special token pooling and uniform mean pooling exhibit biases and undesirable information loss, motivating a broad spectrum of fixed, parametric, attention-based, and learned pooling mechanisms.

1. Fundamental Pooling Operators

The two canonical pooling strategies in modern sequence encoders, especially Transformers, are special token pooling and mean pooling. Given a sequence $X$ of $T$ tokens and the encoder output $$H^\mathrm{enc} = (h_\mathrm{[CLS]}, h_1, \ldots, h_T, h_\mathrm{[SEP]}) \in \mathbb{R}^{(T+2)\times D}$$, the pooling operators are defined as:

• [CLS] Token Pooling: Returns the embedding of the first position, commonly reserved for a pre-inserted “classification signal” token.

$$v_\mathrm{CLS} = \mathcal{P}_\mathrm{CLS}(H^\mathrm{enc}) = h_\mathrm{[CLS]}$$

The surrounding stack must learn, via (global) self-attention, to propagate global features to $h_\mathrm{[CLS]}$. However, rotary position encoding (RoPE) and causal bias result in a marked decay of attention to distant positions, concentrating information toward the prefix and leading to truncation artifacts in long contexts (Doshi et al., 29 Jan 2026).

• Mean Pooling: Computes the arithmetic mean over all token vectors (optionally excluding special tokens or padding).

$$v_\mathrm{mean} = \mathcal{P}_\mathrm{mean}(H^\mathrm{enc}) = \frac{1}{|H^\mathrm{enc}|} \sum_{i=0}^{|H^\mathrm{enc}|-1} h_i$$

Mean pooling is position-agnostic and ensures uniform token contribution, but dilutes salient signals, especially when relevant information is sparse or concentrated (Doshi et al., 29 Jan 2026).

Max pooling, K-max pooling, and softmax-weighted variants, including dynamic (ordinal or learned-weight) pooling, are frequently used in other domains, including multiple-instance learning and weakly supervised localization (Wang et al., 2018, Deliège et al., 2021, Chen et al., 2020). Advanced variants like GPO learn distributions over value orderings, generalizing “hard” max and “soft” mean to optimal pooling strategies discovered end-to-end (Chen et al., 2020).

2. Landmark Pooling: Mechanism and Properties

Landmark (LMK) pooling was introduced to address the contrasting weaknesses of [CLS] bottlenecking and mean pooling dilution, particularly for long contexts in retrieval, reranking, and dense embedding settings (Doshi et al., 29 Jan 2026):

• Partitioning and Token Insertion: The token sequence $X$ is partitioned into $K = \lceil T/C \rceil$ contiguous chunks of size $C$. After each chunk, a special LMK token (typically [SEP]) is inserted, resulting in:

$$X_\mathrm{tok}' = [CLS] \parallel z_1 \parallel [SEP] \parallel z_2 \parallel [SEP] \parallel\ldots\parallel z_K \parallel [SEP]$$

• Learning and Attention: The landmark tokens $\ell_1,\ldots,\ell_K$ are contextually embedded within the full sequence, attending to all real tokens and each other. Their embeddings are updated via standard Transformer self-attention. The LMK tokens are not mere position markers; their parameters are shared with the conventional [SEP] embedding, with position-specific context established through self-attention.
• Aggregation: The final sequence representation is the mean of the $K$ landmark token embeddings:

$$v_\mathrm{LMK} = \mathcal{P}_\mathrm{LMK}(H^\mathrm{enc}) = \frac{1}{K} \sum_{k=1}^{K} h_{\ell_k}$$

This mechanism divides representational capacity across $K$ attention-mediated tokens, distributing the global summarization task while preserving chunk-local information.

• Computational Aspects: Insertion of $K \ll T$ LMK tokens introduces only minor overhead on sequence length (e.g., $<1\%$ for $T=32K$, $C=128$), negligible in terms of $O(S^2 D)$ self-attention cost, with no new model parameters beyond positional embeddings.
• Empirical Results: On standard short-context retrieval benchmarks (MS MARCO, BEIR, MTEB-v2), LMK pooling matches or outperforms [CLS] and mean, with e.g., NDCG@10 scores of LMK = 40.3, CLS = 39.8, and mean = 39.8 on MS MARCO. On long-context tasks (MLDR, COIR, LongEmbed), LMK produces substantial gains: average NDCG@10 of LMK = 48.1 vs. mean = 44.9 and CLS = 37.2 (Doshi et al., 29 Jan 2026).

3. Extensions, Variants, and Comparative Strategies

The spectrum of sequence pooling encompasses fixed, parametric, and fully learned mechanisms:

• Max and Noisy-Or Pooling: Max-pooling identifies the strongest signal per dimension and excels at localization, while noisy-or pooling (from multiple-instance learning) can suppress gradients on correlated sequences, becoming ineffective in highly redundant or temporally clustered signals (Wang et al., 2018). Max-pooling is the preferred strategy for localization-sensitive tasks (e.g., sound event detection, phoneme alignment).
• Eigen Evolution Pooling (EEP): EEP generalizes pooling via PCA of temporal feature trajectories. Each feature sequence is projected onto eigenfunctions (temporal basis vectors) derived from the covariance of all training sequences, preserving higher-order dynamics (oscillations, accelerations) (Wang et al., 2017). EEP outperforms average and rank pooling for fine-grained temporal tasks such as video action recognition.
• Rank Pooling: Introduces a parametric pooling operator based on learning a linear ranking function over frames (i.e., the direction in feature space that best reflects temporal order). The learned vector encodes global sequence evolution (Fernando et al., 2015).
• Hybrid Pooling and Fusion: For architectures capturing multi-scale dependencies (e.g., fusing CNN and RNN features), sequence pooling may be used to merge parallel representations through fixed or learned functions (max, sum, outer-product) (Barbosa et al., 2016).
• Attention-based Pooling: Pooling by Multihead Attention (PMA), as in C2LLM, leverages learnable query tokens to produce sequence embeddings via cross-attention, breaking the causal bottleneck of EOS-based pooling and enabling flexible dimension adaptation. PMA attends directly over all tokens and learns to prioritize relevant subsequences adaptively (Qin et al., 24 Dec 2025).
• Ordinal and GPO Pooling: Trainable weighted pooling over sorted activations (ordinal pooling) or using a learned distribution over the order statistics (GPO) allows the network to interpolate between average and max- (or K-max-) pooling per channel or modality, achieving strong empirical adaptation to retrieval and vision-language embedding tasks (Deliège et al., 2021, Chen et al., 2020).

4. Hierarchical, Multigranular, and Dynamic Pooling

Scaling sequence models to very long contexts, particularly in vision and language, drives the need for pooling-based complexity reduction and multi-resolution summarization:

Model	Pooling Type	Principle	Complexity Reduction
HVT (Pan et al., 2021)	Hierarchical (1D max/avg)	Progressive downsampling	O(N₀²d) → O(∑ₛNₛ²d)
PoNet (Tan et al., 2021)	Multi-granularity (global/segment/local)	Linear instead of quadratic mixing	O(Nd²) vs O(N²d) self-attn
P2T (Wu et al., 2021)	Pyramid Pooling (multi-scale avg)	Multi-ratio pooling × stage	10–100× reduction in N²d
Efficient Transformers (Nawrot et al., 2022)	Dynamic token pooling	Autoregressive learned segmentation	O(l²/SF²) after segmentation

Hierarchical and pyramid pooling in visual transformers (HVT, P2T) insert 1D or 2D pooling operators at block or stage boundaries, reducing token sequence length and enabling reallocation of compute toward depth or width (Pan et al., 2021, Wu et al., 2021). Multi-granular pooling, as in PoNet, fuses global, segmental, and local context via several parallel pooling paths, with lightweight fusion mechanisms (Tan et al., 2021). Adaptive dynamic pooling in autoregressive transformers learns boundary placement (via an MLP predictor or external segmentation), performing mean pooling over variable-length segments, aligning representation to linguistic units and reducing compute (Nawrot et al., 2022).

5. Empirical Comparisons, Ablations, and Trade-Offs

Key ablation and experimental findings clarify the effect of pooling choice:

• Short vs. Long Context: LMK pooling inherits the strong short-context accuracy of [CLS] pooling while overcoming severe degradation as $T \gg$ train context length. CLS pooling’s positional bias is exposed via NDCG@10 metrics: in long-context, CLS drops to 37.2, mean recovers to 44.9, while LMK achieves 48.1 (Doshi et al., 29 Jan 2026).
• Local–Global Tradeoff: Chunk-based LMK pooling preserves fine-grained, chunk-local semantics, as measured by directional Hit@k, while distributing the global summarization burden across $K$ tokens. Increasing the number of landmarks (decreasing chunk size) generally improves long-context extrapolation, with $C \approx 32-64$ optimal (Doshi et al., 29 Jan 2026).
• Pooling Fusion: PoNet ablations confirm that removing any granularity (global, segment, local) impairs downstream performance—sentence-pair tasks are especially harmed by dropping global aggregation; local and segment pooling benefit acceptance and MLM tasks (Tan et al., 2021).
• Hierarchical Pooling in Vision: HVT and P2T demonstrate that global average pooling of visual tokens surpasses class-token pooling for final classification and segmentation accuracy, supporting the move toward nontoken-based, spatially invariant aggregation (Pan et al., 2021, Wu et al., 2021).
• Attention-based Pooling Generality: PMA-enabled models, such as C2LLM, outperform EOS-based and mean pooling in embedding code sequences, supporting scalable and adaptive embedding dimension, with direct impact on code retrieval leaderboards (Qin et al., 24 Dec 2025).
• Hybrid and Learned Pooling: GPO and ordinal pooling outperform hand-tuned K-max and static pooling across vision-language tasks and lightweight CNNs, as the model learns the optimal combination of top-K and uniformly weighted summaries (Chen et al., 2020, Deliège et al., 2021).

6. Applications and Practical Considerations

Sequence pooling is central in applications requiring fixed-size representations from variable-length inputs: dense retrieval, ranking, clustering, action/gait recognition, speech event localization, vision-language embedding, and cross-modal retrieval. Selection between static, attention-based, hierarchical, and learned pooling is task-dependent.

Implementation considerations include tradeoffs in sequence length, model size, attention bias, and computational efficiency:

• Token Overhead: Introducing special tokens (LMK, pooling tokens, PMA queries) increases sequence length modestly, but yields substantial representational benefits (Doshi et al., 29 Jan 2026, Qin et al., 24 Dec 2025).
• Parameter Impact: LMK pooling is parameter-free beyond standard token/position embeddings, whereas PMA and GPO add small MLPs or extra projections but negligible overall cost (Doshi et al., 29 Jan 2026, Qin et al., 24 Dec 2025).
• Contextual Adaptation: Dynamically adapting pooling regions via segmentation predictors or granularity selection enables models to align with natural boundaries—words, phrases, or semantic units—improving both efficiency and downstream accuracy (Nawrot et al., 2022).
• Plug-and-Play Architecture: Many pooling operators (LMK, GPO, PMA) are drop-in replacements in standard Transformer or encoder pipelines, maintaining training stability and compatibility with existing optimizers and losses (Doshi et al., 29 Jan 2026, Chen et al., 2020, Qin et al., 24 Dec 2025).

Sequence pooling has evolved from rigid, stateless reductions to contextually adaptive, end-to-end learned modules that preserve local, global, and hierarchical information. The choice of pooling operator is now a first-class architectural and algorithmic decision in long-context, multimodal, and retrieval-focused models.