---
title: Multi-Level CLS Token Aggregation
url: https://www.emergentmind.com/topics/multi-level-cls-token-aggregation
type: topic
---

# Multi-Level CLS Token Aggregation

Searching arXiv for the cited papers and closely related work on multi-level CLS aggregation.
Multi-level CLS token aggregation denotes a family of representation-learning mechanisms that derive a single prediction or retrieval vector from multiple class-token states, multiple layer summaries, or stage-specific CLS surrogates rather than from a single final-layer [CLS] embedding. Across recent work, the motif appears in multimodal medical vision-language learning, single-pass LLM probing, scientific-document encoding, audio deepfake detection, BERT classification, and CLIP-based dense prediction. The common rationale is that different depths and token positions encode complementary abstractions—fine-grained texture, local evidence, global semantics, or domain-specific signals—and that an explicit aggregation operator can preserve this complementarity in the final readout [2509.00752][2601.13288][2309.04333][2309.08208][2505.15696][2304.06212].

## 1. Conceptual basis

In the ENT endoscopy framework of "Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification" [2509.00752], the Multi-Level Feature Aggregation (MFA) module extracts and fuses the [CLS] tokens from multiple depths of a frozen CLIP ViT-B/16 vision encoder in order to marry fine-grained texture cues from early layers with global semantics from late layers. In "A BERTology View of LLM Orchestrations" [2601.13288], the classification problem is framed as representation selection over the full token-layer hidden-state tensor rather than over a fixed token or fixed layer. In "Encoding Multi-Domain Scientific Papers by Ensembling Multiple CLS Tokens" [2309.04333], multiple prepended CLS tokens are introduced so that each can learn a different document-level aggregation pattern. In "HM-Conformer" [2309.08208], the motivation is that spoofing evidence may exist either locally or globally, so classification tokens are collected from multiple Conformer stages.

A recurrent contrast in this literature is between fixed-CLS readouts and adaptive multi-level readouts. The fixed alternatives include first-token logits, final-layer pooling, or a single standard [CLS] embedding. The cited work repeatedly treats these as sub-optimal whenever the signal is distributed across depths, token positions, domains, or temporal scales [2601.13288][2505.15696][2309.04333].

This suggests that multi-level CLS aggregation is less a single architecture than a readout principle: the encoder is allowed to form multiple intermediate global summaries, and the downstream head learns how to combine them.

## 2. Structural patterns of aggregation

Several distinct architectural patterns recur across the literature. One pattern extracts CLS states from selected depths of a single backbone and fuses them afterward. Another inserts multiple CLS tokens at the input and lets them specialize during the forward pass. A third treats token-wise and layer-wise aggregation as two separate stages. A fourth, adjacent pattern injects or replaces CLS information across multiple shallow layers rather than explicitly collapsing multiple CLS outputs [2509.00752][2601.13288][2309.04333][2304.06212].

| Setting | Aggregation mechanism | Representative paper |
|---|---|---|
| ENT endoscopy | Extract $x_{\mathrm{cls}}^l$ from blocks $4,8,12$; prepend fusion token; fuse with a lightweight Transformer | [2509.00752] |
| Single-pass LLM classification | Aggregate tokens within each layer, then aggregate across layer summaries | [2601.13288] |
| Scientific paper encoding | Prepend $K=3$ CLS tokens; inject per-CLS projections at layers $\{4,8,12\}$; sum final CLS vectors | [2309.04333] |
| Audio deepfake detection | Carry three CLS tokens through six Conformer blocks; extract stage embeddings after blocks $2,4,6$; fuse with a global pooled token | [2309.08208] |
| BERT classification | Max-pool the top-$k$ CLS states, or max-pool token sequences across layers and re-attend with an extra MHA block | [2505.15696] |
| Zero-shot semantic segmentation | Replace the visual CLS token in shallow ViT layers with a projected text CLS token | [2304.06212] |

The ENT endoscopy design is explicitly lightweight: the backbone remains CLIP ViT-B/16, the selected layer count is $K=3$, and the fusion network is a FusionViT with depth $M=2$ small Transformer blocks, $H=12$ heads, $d_{\mathrm{ff}}=3072$, activation $\sigma=\mathrm{GELU}$, and drop-out $0.1$ [2509.00752]. In the LLM-probe setting, the architectural choice spans direct pooling, a scoring-attention gate of approximately $0.1\mathrm{M}$ parameters, and a downcast MHA probe of approximately $35\mathrm{M}$ parameters [2601.13288]. In HM-Conformer, the stage structure is explicit: six Conformer blocks, pooling rate $\gamma=2$, and an initial CLS count that decreases from $3$ to $2$ to $1$ as the network progresses [2309.08208].

A plausible implication is that aggregation depth is usually selected sparsely rather than exhaustively. The cited implementations often prefer a small set of strategically placed levels—such as $\{4,8,12\}$ or three stages—rather than all layers.

## 3. Formal operators and representation geometry

In the endoscopy framework, if the CLIP ViT-B/16 vision encoder has $L$ transformer blocks and $x_{\mathrm{cls}}^l \in \mathbb{R}^d$ denotes the [CLS] output of block $l$, the selected tokens are assembled with a learnable fusion token $t_0$:
$$
\mathcal{T} = [\, t_0; x_{\mathrm{cls}}^{l_1}; x_{\mathrm{cls}}^{l_2}; \dots; x_{\mathrm{cls}}^{l_K} \,] \in \mathbb{R}^{(K+1)\times d}.
$$
A lightweight Transformer then produces the aggregated representation
$$
z_{\mathrm{agg}} = [\mathrm{FusionViT}(\mathcal{T})]_0 \in \mathbb{R}^d.
$$
The internal attention follows the standard form
$$
Q = XW_Q,\quad K = XW_K,\quad V = XW_V,\quad
\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}(QK^\top/\sqrt{d})V,
$$
with FFN and residual connections. The paper also gives an equivalent weighted-sum view:
$$
z_{\mathrm{agg}} = \sigma \Bigl( \sum_{i=1}^K \alpha_i W_i x_{\mathrm{cls}}^{l_i} + W_0 t_0 \Bigr),
$$
where $\sigma(\cdot)$ is GELU and the $\alpha_i$ are implicit attention coefficients learned by the FusionViT [2509.00752].

In the LLM-probe formulation, the hidden state at layer $l$ is $h^{(l)} \in \mathbb{R}^{T\times d}$. A token-level aggregator produces $v^{(l)}=A_{\mathrm{token}}(h^{(l)})$, the layer summaries are stacked into $V\in\mathbb{R}^{L\times d}$, and a second aggregator yields the final classification token $v=A_{\mathrm{layer}}(V)$, followed by $\mathrm{logits}=W_{\mathrm{out}}v+b_{\mathrm{out}}$. The aggregation mechanisms include direct max- or mean-pooling, a scoring-attention gate with $s_i=\tanh(w^\top X_i)$ and $v=\sum_i \alpha_i X_i$, and a downcast multi-head self-attention block followed by pooling over the remaining dimension [2601.13288].

Other papers instantiate different collapse operators. Multi2SPE prepends $K$ learnable CLS tokens, applies per-CLS linear re-parameterization at selected layers, and sums the final vectors:
$$
c_A = \sum_{k=1}^K c_{A,k}.
$$
HM-Conformer collects three stage-specific CLS embeddings and one final SeqPooling vector $g$, then concatenates them and applies a linear projection:
$$
e_5 = \mathrm{concat}(E)W^{(4)} + b^{(4)}.
$$
MaxPoolBERT instead uses depth-wise max-pooling over the top $k$ layers for either the [CLS] token alone or the full $T\times d$ sequence, optionally followed by an extra MHA block in which the final-layer or pooled [CLS] re-attends to all tokens [2309.04333][2309.08208][2505.15696].

These formulations divide naturally into attention-based fusion, additive ensembling, concatenative fusion, and depth-wise pooling. The shared invariant is that the final prediction vector remains $d$-dimensional even though its provenance spans multiple depths or multiple global tokens.

## 4. Coupling to objectives, prompts, and adaptation

Multi-level CLS aggregation is typically not trained in isolation; it is coupled to task-specific objectives that determine what each level should encode. In the ENT endoscopy framework, the aggregated vector $z_{\mathrm{agg}}$ is LayerNorm-ed and then $\ell_2$-normalized:
$$
v = z_{\mathrm{agg}}/\|z_{\mathrm{agg}}\|_2.
$$
A frozen CLIP text encoder produces the matching text embedding $u\in\mathbb{R}^d$ for each class-prompt, and training uses a symmetric CLIP-style InfoNCE loss with a learnable temperature $\tau$. The framework also combines supervised classification with contrastive learning, uses class-specific natural language prompts, applies Low-Rank Adaptation within every ViT block of the image encoder, and augments same-class normalized aggregated embeddings via spherical feature interpolation using Slerp [2509.00752].

In Multi2SPE, training uses citation-prediction cross-entropy over positive and negative citation pairs, but similarity is made multi-CLS-aware:
$$
S_{\mathrm{MC}}(Q,P)=\lambda \max_{i,j}(c_{Q,i}\cdot c_{P,j}) + (1-\lambda)(c_Q\cdot c_P).
$$
The max term pushes at least one pair of sub-embeddings to align strongly, while the sum term encourages the ensemble to work as a whole; the paper reports that $\lambda=0.1$ gave the best results [2309.04333].

HM-Conformer attaches auxiliary OC-Softmax heads not only to the final fused embedding but also to each stage-specific CLS token and the global token. This auxiliary supervision is described as encouraging each CLS token to learn stage-appropriate, task-relevant features. The loss weights for the five heads are typically set to $4:3:2:1:1$, while inference uses only the final score derived from the fused embedding [2309.08208].

In the LLM-probe setting, the objective is operationally different but structurally related: a lightweight classifier is trained on hidden states that are already produced by the serving LLM, so labels are predicted in the same forward pass used for generation. The aggregation mechanism is therefore a readout over a frozen model’s hidden-state tensor rather than a redesign of the base model itself [2601.13288].

An adjacent variant appears in ClsCLIP. There, the text-side [CLS] token is treated as a category prior and is used to replace the visual [CLS] token in shallow ViT layers from $N_1$ to $N_2-1$, after which the patch tokens are decoded for segmentation. The method does not aggregate multiple extracted CLS outputs into a single vector, but it is still multi-level in the sense that the same CLS prior is injected across multiple consecutive layers [2304.06212].

## 5. Empirical record across domains

The empirical literature consistently reports that multi-level CLS aggregation improves over single-level or fixed-readout baselines, although the magnitude depends on domain, backbone, and aggregation operator [2509.00752][2601.13288][2309.04333][2309.08208][2505.15696].

| Paper | Comparison | Reported outcome |
|---|---|---|
| [2509.00752] | LoRA only $\rightarrow$ LoRA + MFA | Accuracy $0.88 \rightarrow 0.93$; Recall@1 (i2i) $0.89 \rightarrow 0.91$ |
| [2509.00752] | Full model | $95\%$ accuracy and F1-score; Recall@1 $0.93$ (i2i), $0.92$ (t2i); MRR $0.97$ and $0.96$ |
| [2601.13288] | Direct pooling $\rightarrow$ scoring gate $\rightarrow$ MHA | ToxicChat F1 (ID) $73.5\% \rightarrow 80.5\% \rightarrow 84.5\%$; WildGuard F1 $82.8\% \rightarrow 86.0\% \rightarrow 88.6\%$ |
| [2309.04333] | Multi2SPE vs baselines | Up to $25$ percent relative error reduction in multi-domain citation prediction |
| [2309.08208] | Partial subsets vs all stage tokens | EER improves to $15.71\%$ when all $\{$stage 1,2,3, global$\}$ are used |
| [2505.15696] | BERT-base vs MaxPoolBERT | GLUE average $79.63 \rightarrow 80.88$ |

The endoscopy paper supplies a particularly explicit ablation of MFA. On the public ENTRep test split, the baseline without LoRA, MFA, or SFA reaches accuracy $0.84$, image-to-image Recall@1 $0.85$, and text-to-image Recall@1 $0.84$; adding LoRA raises these to $0.88$, $0.89$, and $0.90$; adding MFA gives $0.93$, $0.91$, and $0.92$; and the full model with LoRA, MFA, and SFA reaches $0.95$, $0.93$, and $0.92$ [2509.00752].

The LLM-probe paper makes the same point with a different baseline family. First-token logits (MULI) reach ToxicChat F1 $77.8\%$, WildGuard $83.8\%$, IMDB $86.5\%$, SST-2 $93.19\%$, and Emotion $64.05\%$, while multi-level probes improve on these scores, with the ranking pooling $<$ scoring $<$ MHA reported as consistent across safety and sentiment tasks [2601.13288].

Multi2SPE reports that $K=3$ is the default and that $K=5$ is comparable to $K=3$ within $\pm 1\%$ error, while $K=1$ causes a performance drop. The ablations also report that removing all linear layers causes $-3\%$ more error and removing the re-parameterization causes $-2.5\%$ more error [2309.04333]. HM-Conformer provides an especially direct aggregation ablation: only stage 1 + global gives EER $17.41\%$, only stage 2 + global gives $16.59\%$, only stage 3 + global gives $17.38\%$, stages 1 + 3 + global give $16.92\%$, stages 2 + 3 + global give $17.74\%$, stages 1 + 2 + 3 without the global token give $16.71\%$, and all four tokens give $15.71\%$ [2309.08208].

## 6. Efficiency, design trade-offs, and common misunderstandings

A common misunderstanding is that multi-level CLS aggregation necessarily implies a second large model or a large increase in serving cost. The evidence in the cited work is more differentiated. In the LLM-probe setting, direct pooling introduces $0$ parameters, the scoring-attention gate uses approximately $(L+1)d \approx 92\mathrm{K}\approx 0.1\mathrm{M}$ parameters, and the downcast MHA probe uses approximately $35\mathrm{M}$ parameters; even the MHA variant remains single-pass and is reported as far smaller in latency and memory cost than a Guard$\rightarrow$Serve pipeline with an extra $8\mathrm{B}$ model [2601.13288]. In MaxPoolBERT, max-pooling adds no parameters, and the extra MHA block adds approximately $2.36\mathrm{M}$ parameters, or approximately $2.1\%$ over BERT-base’s approximately $110\mathrm{M}$ parameters [2505.15696]. Multi2SPE is described as requiring only a negligible amount of computation in addition to one BERT forward pass [2309.04333].

The efficiency tables in the LLM-probe paper make the trade-off explicit. On $1$-token safety inference with Llama-3.2-3B, throughput and latency are reported as $37.8$ s/s and $26.4$ ms for the base model, $33.7$ s/s and $29.7$ ms with direct pooling, $32.4$ s/s and $30.9$ ms with the scoring gate, $24.8$ s/s and $40.3$ ms with MHA downcast, and $8.1$ s/s and $123.2$ ms for Guard$\rightarrow$Serve $(8\mathrm{B}+3\mathrm{B})$; peak GPU memory is $6{,}498$, $6{,}498$, $6{,}749$, $6{,}969$, and $22{,}769$ MB respectively [2601.13288].

Another misunderstanding is that “multi-level” always means “use all layers.” The cited designs do not support that view. The endoscopy framework chooses blocks $4,8,12$ out of a $12$-block ViT; Multi2SPE injects specialization at the same three layers; HM-Conformer defines three stages over six blocks; and MaxPoolBERT reports that $k=3$ is near-optimal across CoLA, MRPC, and RTE [2509.00752][2309.04333][2309.08208][2505.15696].

A further point of clarification concerns terminology. Some methods aggregate multiple extracted CLS vectors into one output vector; others create several CLS tokens that specialize jointly; others summarize tokens first and layers second; and ClsCLIP performs repeated CLS replacement rather than post hoc CLS fusion. The family resemblance is therefore functional rather than strictly architectural: all of these methods treat the CLS channel as a manipulable, multi-level interface for collecting task-relevant global information [2304.06212].

Taken together, the literature supports a narrow but robust conclusion. When task evidence is distributed across layers, positions, domains, or stages, a learned multi-level CLS readout often outperforms fixed single-level alternatives, and it can do so with anything from parameter-free pooling to modestly sized attention modules, depending on the operating point required by the application [2509.00752][2601.13288][2505.15696].

Source: https://www.emergentmind.com/topics/multi-level-cls-token-aggregation