Multi-Level CLS Token Aggregation

• Multi-Level CLS Token Aggregation is a technique that fuses outputs from selected layers to capture both fine-grained textures and global semantics.
• It is applied across diverse domains such as medical vision-language tasks, document encoding, and audio deepfake detection, leveraging complementary token features.
• Empirical studies consistently show that this adaptive aggregation approach outperforms fixed single-level readouts, improving accuracy and reducing errors.

Searching arXiv for the cited papers and closely related work on multi-level CLS aggregation. Multi-level CLS token aggregation denotes a family of representation-learning mechanisms that derive a single prediction or retrieval vector from multiple class-token states, multiple layer summaries, or stage-specific CLS surrogates rather than from a single final-layer [CLS] embedding. Across recent work, the motif appears in multimodal medical vision-language learning, single-pass LLM probing, scientific-document encoding, audio deepfake detection, BERT classification, and CLIP-based dense prediction. The common rationale is that different depths and token positions encode complementary abstractions—fine-grained texture, local evidence, global semantics, or domain-specific signals—and that an explicit aggregation operator can preserve this complementarity in the final readout (Nguyen et al., 31 Aug 2025, Meyoyan et al., 19 Jan 2026, Seoh et al., 2023, Shin et al., 2023, Behrendt et al., 21 May 2025, Wu et al., 2023).

1. Conceptual basis

In the ENT endoscopy framework of "Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification" (Nguyen et al., 31 Aug 2025), the Multi-Level Feature Aggregation (MFA) module extracts and fuses the [CLS] tokens from multiple depths of a frozen CLIP ViT-B/16 vision encoder in order to marry fine-grained texture cues from early layers with global semantics from late layers. In "A BERTology View of LLM Orchestrations" (Meyoyan et al., 19 Jan 2026), the classification problem is framed as representation selection over the full token-layer hidden-state tensor rather than over a fixed token or fixed layer. In "Encoding Multi-Domain Scientific Papers by Ensembling Multiple CLS Tokens" (Seoh et al., 2023), multiple prepended CLS tokens are introduced so that each can learn a different document-level aggregation pattern. In "HM-Conformer" (Shin et al., 2023), the motivation is that spoofing evidence may exist either locally or globally, so classification tokens are collected from multiple Conformer stages.

A recurrent contrast in this literature is between fixed-CLS readouts and adaptive multi-level readouts. The fixed alternatives include first-token logits, final-layer pooling, or a single standard [CLS] embedding. The cited work repeatedly treats these as sub-optimal whenever the signal is distributed across depths, token positions, domains, or temporal scales (Meyoyan et al., 19 Jan 2026, Behrendt et al., 21 May 2025, Seoh et al., 2023).

This suggests that multi-level CLS aggregation is less a single architecture than a readout principle: the encoder is allowed to form multiple intermediate global summaries, and the downstream head learns how to combine them.

2. Structural patterns of aggregation

Several distinct architectural patterns recur across the literature. One pattern extracts CLS states from selected depths of a single backbone and fuses them afterward. Another inserts multiple CLS tokens at the input and lets them specialize during the forward pass. A third treats token-wise and layer-wise aggregation as two separate stages. A fourth, adjacent pattern injects or replaces CLS information across multiple shallow layers rather than explicitly collapsing multiple CLS outputs (Nguyen et al., 31 Aug 2025, Meyoyan et al., 19 Jan 2026, Seoh et al., 2023, Wu et al., 2023).

Setting	Aggregation mechanism	Representative paper
ENT endoscopy	Extract $x_{\mathrm{cls}}^l$ from blocks $4,8,12$; prepend fusion token; fuse with a lightweight Transformer	(Nguyen et al., 31 Aug 2025)
Single-pass LLM classification	Aggregate tokens within each layer, then aggregate across layer summaries	(Meyoyan et al., 19 Jan 2026)
Scientific paper encoding	Prepend $K=3$ CLS tokens; inject per-CLS projections at layers $\{4,8,12\}$; sum final CLS vectors	(Seoh et al., 2023)
Audio deepfake detection	Carry three CLS tokens through six Conformer blocks; extract stage embeddings after blocks $2,4,6$; fuse with a global pooled token	(Shin et al., 2023)
BERT classification	Max-pool the top-$k$ CLS states, or max-pool token sequences across layers and re-attend with an extra MHA block	(Behrendt et al., 21 May 2025)
Zero-shot semantic segmentation	Replace the visual CLS token in shallow ViT layers with a projected text CLS token	(Wu et al., 2023)

The ENT endoscopy design is explicitly lightweight: the backbone remains CLIP ViT-B/16, the selected layer count is $K=3$, and the fusion network is a FusionViT with depth $M=2$ small Transformer blocks, $H=12$ heads, $d_{\mathrm{ff}}=3072$, activation $4,8,12$0, and drop-out $4,8,12$1 (Nguyen et al., 31 Aug 2025). In the LLM-probe setting, the architectural choice spans direct pooling, a scoring-attention gate of approximately $4,8,12$2 parameters, and a downcast MHA probe of approximately $4,8,12$3 parameters (Meyoyan et al., 19 Jan 2026). In HM-Conformer, the stage structure is explicit: six Conformer blocks, pooling rate $4,8,12$4, and an initial CLS count that decreases from $4,8,12$5 to $4,8,12$6 to $4,8,12$7 as the network progresses (Shin et al., 2023).

A plausible implication is that aggregation depth is usually selected sparsely rather than exhaustively. The cited implementations often prefer a small set of strategically placed levels—such as $4,8,12$8 or three stages—rather than all layers.

3. Formal operators and representation geometry

In the endoscopy framework, if the CLIP ViT-B/16 vision encoder has $4,8,12$9 transformer blocks and $K=3$0 denotes the [CLS] output of block $K=3$1, the selected tokens are assembled with a learnable fusion token $K=3$2:

$K=3$3

A lightweight Transformer then produces the aggregated representation

$K=3$4

The internal attention follows the standard form

$K=3$5

with FFN and residual connections. The paper also gives an equivalent weighted-sum view:

$K=3$6

where $K=3$7 is GELU and the $K=3$8 are implicit attention coefficients learned by the FusionViT (Nguyen et al., 31 Aug 2025).

In the LLM-probe formulation, the hidden state at layer $K=3$9 is $\{4,8,12\}$0. A token-level aggregator produces $\{4,8,12\}$1, the layer summaries are stacked into $\{4,8,12\}$2, and a second aggregator yields the final classification token $\{4,8,12\}$3, followed by $\{4,8,12\}$4. The aggregation mechanisms include direct max- or mean-pooling, a scoring-attention gate with $\{4,8,12\}$5 and $\{4,8,12\}$6, and a downcast multi-head self-attention block followed by pooling over the remaining dimension (Meyoyan et al., 19 Jan 2026).

Other papers instantiate different collapse operators. Multi2SPE prepends $\{4,8,12\}$7 learnable CLS tokens, applies per-CLS linear re-parameterization at selected layers, and sums the final vectors:

$\{4,8,12\}$8

HM-Conformer collects three stage-specific CLS embeddings and one final SeqPooling vector $\{4,8,12\}$9, then concatenates them and applies a linear projection:

$2,4,6$0

MaxPoolBERT instead uses depth-wise max-pooling over the top $2,4,6$1 layers for either the [CLS] token alone or the full $2,4,6$2 sequence, optionally followed by an extra MHA block in which the final-layer or pooled [CLS] re-attends to all tokens (Seoh et al., 2023, Shin et al., 2023, Behrendt et al., 21 May 2025).

These formulations divide naturally into attention-based fusion, additive ensembling, concatenative fusion, and depth-wise pooling. The shared invariant is that the final prediction vector remains $2,4,6$3-dimensional even though its provenance spans multiple depths or multiple global tokens.

4. Coupling to objectives, prompts, and adaptation

Multi-level CLS aggregation is typically not trained in isolation; it is coupled to task-specific objectives that determine what each level should encode. In the ENT endoscopy framework, the aggregated vector $2,4,6$4 is LayerNorm-ed and then $2,4,6$5-normalized:

$2,4,6$6

A frozen CLIP text encoder produces the matching text embedding $2,4,6$7 for each class-prompt, and training uses a symmetric CLIP-style InfoNCE loss with a learnable temperature $2,4,6$8. The framework also combines supervised classification with contrastive learning, uses class-specific natural language prompts, applies Low-Rank Adaptation within every ViT block of the image encoder, and augments same-class normalized aggregated embeddings via spherical feature interpolation using Slerp (Nguyen et al., 31 Aug 2025).

In Multi2SPE, training uses citation-prediction cross-entropy over positive and negative citation pairs, but similarity is made multi-CLS-aware:

$2,4,6$9

The max term pushes at least one pair of sub-embeddings to align strongly, while the sum term encourages the ensemble to work as a whole; the paper reports that $k$0 gave the best results (Seoh et al., 2023).

HM-Conformer attaches auxiliary OC-Softmax heads not only to the final fused embedding but also to each stage-specific CLS token and the global token. This auxiliary supervision is described as encouraging each CLS token to learn stage-appropriate, task-relevant features. The loss weights for the five heads are typically set to $k$1, while inference uses only the final score derived from the fused embedding (Shin et al., 2023).

In the LLM-probe setting, the objective is operationally different but structurally related: a lightweight classifier is trained on hidden states that are already produced by the serving LLM, so labels are predicted in the same forward pass used for generation. The aggregation mechanism is therefore a readout over a frozen model’s hidden-state tensor rather than a redesign of the base model itself (Meyoyan et al., 19 Jan 2026).

An adjacent variant appears in ClsCLIP. There, the text-side [CLS] token is treated as a category prior and is used to replace the visual [CLS] token in shallow ViT layers from $k$2 to $k$3, after which the patch tokens are decoded for segmentation. The method does not aggregate multiple extracted CLS outputs into a single vector, but it is still multi-level in the sense that the same CLS prior is injected across multiple consecutive layers (Wu et al., 2023).

5. Empirical record across domains

The empirical literature consistently reports that multi-level CLS aggregation improves over single-level or fixed-readout baselines, although the magnitude depends on domain, backbone, and aggregation operator (Nguyen et al., 31 Aug 2025, Meyoyan et al., 19 Jan 2026, Seoh et al., 2023, Shin et al., 2023, Behrendt et al., 21 May 2025).

Paper	Comparison	Reported outcome
(Nguyen et al., 31 Aug 2025)	LoRA only $k$4 LoRA + MFA	Accuracy $k$5; Recall@1 (i2i) $k$6
(Nguyen et al., 31 Aug 2025)	Full model	$k$7 accuracy and F1-score; Recall@1 $k$8 (i2i), $k$9 (t2i); MRR $K=3$0 and $K=3$1
(Meyoyan et al., 19 Jan 2026)	Direct pooling $K=3$2 scoring gate $K=3$3 MHA	ToxicChat F1 (ID) $K=3$4; WildGuard F1 $K=3$5
(Seoh et al., 2023)	Multi2SPE vs baselines	Up to $K=3$6 percent relative error reduction in multi-domain citation prediction
(Shin et al., 2023)	Partial subsets vs all stage tokens	EER improves to $K=3$7 when all $K=3$8stage 1,2,3, global$K=3$9 are used
(Behrendt et al., 21 May 2025)	BERT-base vs MaxPoolBERT	GLUE average $M=2$0

The endoscopy paper supplies a particularly explicit ablation of MFA. On the public ENTRep test split, the baseline without LoRA, MFA, or SFA reaches accuracy $M=2$1, image-to-image Recall@1 $M=2$2, and text-to-image Recall@1 $M=2$3; adding LoRA raises these to $M=2$4, $M=2$5, and $M=2$6; adding MFA gives $M=2$7, $M=2$8, and $M=2$9; and the full model with LoRA, MFA, and SFA reaches $H=12$0, $H=12$1, and $H=12$2 (Nguyen et al., 31 Aug 2025).

The LLM-probe paper makes the same point with a different baseline family. First-token logits (MULI) reach ToxicChat F1 $H=12$3, WildGuard $H=12$4, IMDB $H=12$5, SST-2 $H=12$6, and Emotion $H=12$7, while multi-level probes improve on these scores, with the ranking pooling $H=12$8 scoring $H=12$9 MHA reported as consistent across safety and sentiment tasks (Meyoyan et al., 19 Jan 2026).

Multi2SPE reports that $d_{\mathrm{ff}}=3072$0 is the default and that $d_{\mathrm{ff}}=3072$1 is comparable to $d_{\mathrm{ff}}=3072$2 within $d_{\mathrm{ff}}=3072$3 error, while $d_{\mathrm{ff}}=3072$4 causes a performance drop. The ablations also report that removing all linear layers causes $d_{\mathrm{ff}}=3072$5 more error and removing the re-parameterization causes $d_{\mathrm{ff}}=3072$6 more error (Seoh et al., 2023). HM-Conformer provides an especially direct aggregation ablation: only stage 1 + global gives EER $d_{\mathrm{ff}}=3072$7, only stage 2 + global gives $d_{\mathrm{ff}}=3072$8, only stage 3 + global gives $d_{\mathrm{ff}}=3072$9, stages 1 + 3 + global give $4,8,12$00, stages 2 + 3 + global give $4,8,12$01, stages 1 + 2 + 3 without the global token give $4,8,12$02, and all four tokens give $4,8,12$03 (Shin et al., 2023).

6. Efficiency, design trade-offs, and common misunderstandings

A common misunderstanding is that multi-level CLS aggregation necessarily implies a second large model or a large increase in serving cost. The evidence in the cited work is more differentiated. In the LLM-probe setting, direct pooling introduces $4,8,12$04 parameters, the scoring-attention gate uses approximately $4,8,12$05 parameters, and the downcast MHA probe uses approximately $4,8,12$06 parameters; even the MHA variant remains single-pass and is reported as far smaller in latency and memory cost than a Guard$4,8,12$07Serve pipeline with an extra $4,8,12$08 model (Meyoyan et al., 19 Jan 2026). In MaxPoolBERT, max-pooling adds no parameters, and the extra MHA block adds approximately $4,8,12$09 parameters, or approximately $4,8,12$10 over BERT-base’s approximately $4,8,12$11 parameters (Behrendt et al., 21 May 2025). Multi2SPE is described as requiring only a negligible amount of computation in addition to one BERT forward pass (Seoh et al., 2023).

The efficiency tables in the LLM-probe paper make the trade-off explicit. On $4,8,12$12-token safety inference with Llama-3.2-3B, throughput and latency are reported as $4,8,12$13 s/s and $4,8,12$14 ms for the base model, $4,8,12$15 s/s and $4,8,12$16 ms with direct pooling, $4,8,12$17 s/s and $4,8,12$18 ms with the scoring gate, $4,8,12$19 s/s and $4,8,12$20 ms with MHA downcast, and $4,8,12$21 s/s and $4,8,12$22 ms for Guard$4,8,12$23Serve $4,8,12$24; peak GPU memory is $4,8,12$25, $4,8,12$26, $4,8,12$27, $4,8,12$28, and $4,8,12$29 MB respectively (Meyoyan et al., 19 Jan 2026).

Another misunderstanding is that “multi-level” always means “use all layers.” The cited designs do not support that view. The endoscopy framework chooses blocks $4,8,12$30 out of a $4,8,12$31-block ViT; Multi2SPE injects specialization at the same three layers; HM-Conformer defines three stages over six blocks; and MaxPoolBERT reports that $4,8,12$32 is near-optimal across CoLA, MRPC, and RTE (Nguyen et al., 31 Aug 2025, Seoh et al., 2023, Shin et al., 2023, Behrendt et al., 21 May 2025).

A further point of clarification concerns terminology. Some methods aggregate multiple extracted CLS vectors into one output vector; others create several CLS tokens that specialize jointly; others summarize tokens first and layers second; and ClsCLIP performs repeated CLS replacement rather than post hoc CLS fusion. The family resemblance is therefore functional rather than strictly architectural: all of these methods treat the CLS channel as a manipulable, multi-level interface for collecting task-relevant global information (Wu et al., 2023).

Taken together, the literature supports a narrow but robust conclusion. When task evidence is distributed across layers, positions, domains, or stages, a learned multi-level CLS readout often outperforms fixed single-level alternatives, and it can do so with anything from parameter-free pooling to modestly sized attention modules, depending on the operating point required by the application (Nguyen et al., 31 Aug 2025, Meyoyan et al., 19 Jan 2026, Behrendt et al., 21 May 2025).