Shared Aggregation Tokens: Architectures & Applications

• Shared Aggregation Tokens are reusable carriers that combine inputs from various sources to drive control, prediction, fusion, and retrieval in diverse systems.
• They improve parameter and communication efficiency by compressing representations and maintaining fixed interfaces across modalities and domains.
• However, careful design is essential since shared pathways can lead to inseparable control-data mixing, raising vulnerabilities like prompt injection.

Shared Aggregation Tokens are token-like entities whose state is formed by aggregating information from multiple sources and then reused on a downstream computation path. Across recent arXiv literature, the phrase is attached to several non-identical mechanisms: control-relevant nodes in shared-embedding sequence models, compositional token embeddings built from shared concept vectors, proxy tokens that preserve a modality-complete fusion interface, learnable aggregation tokens inside vision transformers, shared retrieval prototypes for cross-view place recognition, cross-example mixed tokens in continuous mixture-of-experts, mobile aggregation carriers in distributed networks, and uncoded groupwise keys in secure aggregation (Pant et al., 25 Jun 2026, V et al., 22 Sep 2025, Wang et al., 2 Apr 2026, Lu et al., 8 Nov 2025, Xu et al., 10 May 2026, Antoniak et al., 2023, Wan et al., 2022). This suggests a broad architectural pattern rather than a single canonical object: a shared aggregation token is a reusable carrier, slot, or prototype whose effect depends on many inputs while remaining on a common control, prediction, or communication path.

1. Scope and definitional variants

Across the cited literature, the same label is used for different technical roles. In some works, the token is a literal sequence element processed by attention; in others, it is a mobile state carrier, a retrieval prototype, or a shared key. The commonality is aggregation plus reuse: multiple sources contribute to a shared representation, and that representation subsequently governs classification, control, fusion, retrieval, or communication.

Domain	Meaning of “Shared Aggregation Tokens”	Representative paper
Shared-embedding LLM security	Control-relevant nodes whose hidden states aggregate trusted and untrusted tokens	(Pant et al., 25 Jun 2026)
Embedding compression	Token embeddings composed from shared Concept Vectors	(V et al., 22 Sep 2025)
Missing-modality fusion	Replacement tokens aggregated from source-specific proxies	(Wang et al., 2 Apr 2026)
Transformer VPR	Learnable aggregation tokens prepended to patch tokens	(Lu et al., 8 Nov 2025)
Cross-view retrieval	Shared VLAQ query centers acting as retrieval prototypes	(Xu et al., 10 May 2026)
Continuous MoE	Per-expert mixed tokens aggregated across examples	(Antoniak et al., 2023)
Distributed aggregation	Mobile tokens carrying partial aggregates through a network	(Salehkaleybar et al., 2017)
Secure aggregation	Uncoded groupwise shared keys used for masking	(Wan et al., 2022)

A second cross-cutting distinction concerns what is shared. Some systems share parameters across tokens, as in Aggregate Semantic Grouping and VLAQ query centers. Others share state across data instances, as in Mixture of Tokens or distributed token protocols. Still others share control authority, which is precisely what becomes problematic in shared-embedding prompted systems: the same aggregation pathway carries both instructions and untrusted data into control-authoritative outputs (Pant et al., 25 Jun 2026).

2. Control-relevant aggregation in shared-embedding sequence models

In the security analysis of prompted systems, the phrase “Shared Aggregation Tokens” is not used explicitly; the closest mechanisms are the shared-embedding tokens and the control-relevant aggregator nodes whose hidden states are formed by attention value aggregation and then read out by control heads for refusal, tool authorization, routing, and memory-write authorization (Pant et al., 25 Jun 2026). The formal model is a Prompted Action Model $\mathcal{M}=(\Sigma,d,E,f,\mathcal{A})$ in which trusted instructions $s$ and untrusted user input $u$ are concatenated, embedded by a single embedding function $E$, and processed by the same attention-residual pipeline. Control-authoritative actions are part of the output space.

The central property is Semantic-Faithful Control (SFC): control-authoritative behavior should depend only on the semantic content of untrusted input, not on semantically irrelevant representational variations. The paper shows that SFC is unachievable within the shared pipeline through three results. First, provenance recovery is impossible in general because trusted and untrusted content become statistically inseparable in shared representations, with Bayes-optimal provenance error bounded away from zero by total variation distance. Second, control-path exposure follows whenever a control-relevant node assigns nonzero attention mass to untrusted positions; then untrusted values enter the very computation that influences control decisions. Third, finite training or alignment cannot certify invariance over infinite semantic-equivalence classes such as paraphrases, Unicode or homoglyph substitutions, base64 encodings, semantic-vacuous suffixes, and formatting changes (Pant et al., 25 Jun 2026).

The empirical grounding is intended to show that the argument is structural rather than merely asymptotic. Across Mistral-7B, Llama-3.1, and Qwen-2.5 tokenizers, instruction coverage $\rho_{\text{cov}}$ ranges 61.5%–82.3%, indicating extensive representational collision between instruction and user data. Hidden-state distributions for instruction versus user content remain substantially overlapping at all layers, with MMD approximately 0.61 at layer 0 and approximately 0.76 at layer 31. Behavioral experiments further show that prompt injections can change structural output properties such as length, preambles, and framing even when refusal is preserved, and “abliteration weight surgery” demonstrates that a low-rank refusal direction can be removed from residual streams without retraining, reducing refusal broadly across inputs (Pant et al., 25 Jun 2026).

The implication is that in-pipeline defenses such as alignment, classifiers, perplexity filters, circuit breakers, positional markers, or segment delimiters cannot guarantee perfect prompt-injection prevention. The architectural remedies proposed are separation mechanisms: distinct embedding spaces and disjoint encoders for trusted and untrusted channels, typed attention or hard segment masks that enforce $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$ for control paths, separate control readout stacks, external policy engines, and immutable provenance tags carried through the network from the first layer. This aligns the paper’s security analogy to Von Neumann code-data confusion: the vulnerability arises from shared control-data pathways, not from insufficient tuning of a classifier inside the same pathway (Pant et al., 25 Jun 2026).

3. Compositional token representations and shared semantic building blocks

In embedding compression, Shared Aggregation Tokens are not control nodes but compositional representations. Aggregate Semantic Grouping (ASG) treats a token embedding not as a single monolithic vector but as an aggregation of shared semantic building blocks, called Concept Vectors, learned through Product Quantization (V et al., 22 Sep 2025). The original embedding space of dimension $d$ is divided into $M$ subspaces; for each subspace, a codebook $C_m\in\mathbb{R}^{(d/M)\times K}$ is learned by k-means over pretrained token embeddings, and each token is represented by one ConceptID per subspace. In the reported experiments, the token representation is the concatenation of the selected Concept Vectors.

This architecture changes the scaling law of the embedding layer. Standard embeddings have $Vd$ trainable parameters, while ASG codebooks have $s$0 parameters, independent of $s$1, so the embedding compression ratio is $s$2. For XLM-R with $s$3, $s$4, $s$5, and $s$6, the embedding layer is reduced to approximately 0.4% of the original size; for mBERT with $s$7, $s$8, $s$9, and $u$0, the reduction is approximately 0.50% (V et al., 22 Sep 2025). The token-to-ConceptID mapping is stored separately as a fixed integer matrix and is reported as “not a parameter.”

The reported empirical claim is that this extreme compression preserves most downstream performance. On XNLI and WikiANN, mBERT ASG at 0.50% embedding size attains 73.51 accuracy on XNLI and 88.11 F1 on WikiANN, corresponding to 97.42% and 98.19% of the baseline. XLM-R ASG at 0.40% embedding size attains 77.06 on XNLI and 86.53 on WikiANN, corresponding to 98.81% and 97.92% of the baseline. In mT5 generative settings, the shared-codebook variant with $u$1 and $u$2 reaches 0.25% embedding size while reporting TyDiQA F1 70.81 and WikiANN F1 84.19, or 100.09% and 99.97% of the baseline, with zero-shot WikiANN F1 47.23, or 93.06% of baseline (V et al., 22 Sep 2025).

The qualitative interpretation is that different subspaces can encode different semantic facets. The paper’s “father” example clusters one subspace with kinship terms, another with religious or authority terms, and another with mythic or abstract associations. This supports the claim that compositionality can handle polysemy better than monolithic embeddings. The same paper, however, also reports clear capacity limits: too few subspaces, such as $u$3, degrade performance, while very large $u$4 makes each subspace dimension tiny and increases ConceptID sequence length. Larger codebooks improve quantization error and are especially helpful in generative and zero-shot settings (V et al., 22 Sep 2025).

4. Vision and multimodal perception

In multimodal sensing with missing modalities, COMPASS uses Shared Aggregation Tokens as slot-filling replacement tokens. The framework enforces a fixed $u$5-slot multimodal interface in which the fusion head always receives one token per modality slot. If a modality is missing, COMPASS generates directed source-to-target proxy tokens $u$6 from each observed source and aggregates them into a single replacement token by uniform averaging, $u$7 (Wang et al., 2 Apr 2026). The generators are single-layer Transformer encoders with $u$8 attention heads, operating in a shared latent space with $u$9 and $E$0. The training objective combines task loss, proxy alignment, VICReg shared-space regularization, and per-proxy discriminative supervision, with $E$1, $E$2, and $E$3. On XRF55, COMPASS reports a 7-scenario average of 84.6% versus 72.1% for X-Fi; on MM-Fi, a 15-combination average of 70.6% versus 62.5% for X-Fi; and on OctoNet, wins or ties in 30 of 31 combinations (Wang et al., 2 Apr 2026). The design significance is that aggregation preserves the fusion interface rather than adapting the fusion head to every observed subset.

In visual place recognition, ImAge treats Shared Aggregation Tokens as learnable global parameters prepended to the ViT token stream before a specific transformer block. These tokens are shared across all inputs, inserted once before the fourth-to-last block of a DINOv2-base-register backbone, and then carried through the remaining trainable blocks (Lu et al., 8 Nov 2025). The final descriptor is formed by concatenating the last-layer aggregation tokens and L2-normalizing the result. With $E$4 tokens and token dimension $E$5, the descriptor is 6144-dimensional. The recommended initialization uses L2-normalized k-means centers computed from sampled backbone descriptors. The method reports Pitts30k R@1 of 94.1, MSLS-val R@1 of 94.5, Tokyo24/7 R@1 of 97.1, Nordland R@1 of 97.7, and rank 1 on the MSLS challenge leaderboard. It also reports 14.8 ms inference time, compared with 15.0 ms for NetVLAD, 16.3 ms for SALAD, and 16.4 ms for BoQ, while using approximately 0.006M parameters for the aggregation tokens themselves (Lu et al., 8 Nov 2025). The paper’s main claim is that the backbone’s own self-attention can act as an implicit aggregator, eliminating a separate aggregation head.

In multimodal aerial-ground cross-view place recognition, MAG-VLAQ uses shared VLAQ query centers $E$6 as retrieval prototypes. Aerial descriptors are aggregated directly with this shared prototype set, while ground RGB-LiDAR descriptors use observation-adapted prototypes $E$7, where $E$8 is predicted from a fused multimodal state produced by neural ODE blocks (Xu et al., 10 May 2026). The VLAQ mechanism aggregates token-to-query residuals, and the query-shift regularizer $E$9 keeps adapted centers close to the shared prototypes. The system uses DINOv2 for aerial and ground images, Utonia for LiDAR, and $\rho_{\text{cov}}$0 learnable query prototypes, producing 512-dimensional L2-normalized descriptors. On KITTI360-AG in the satellite setting, MAG-VLAQ reports 61.1 Recall@1 versus 34.5 for ConGeo and 32.0 for AGPlace; on nuScenes-AG with cameras plus LiDAR, it reports 82.6 Recall@1 versus 75.6 for AGPlace, and it remains substantially stronger under camera failure and LiDAR failure (Xu et al., 10 May 2026). Here the shared aggregation token is not a sequence token in the usual sense but a modality- and view-agnostic retrieval prototype.

A different multimodal fusion use appears in the 1D-token image-fusion interface built on a frozen TiTok tokenizer. There, the 1D token sequence $\rho_{\text{cov}}$1 acts as a shared global carrier of appearance or base factors across modalities, and Selective Token Editing updates only a small set of critical positions and channels by $\rho_{\text{cov}}$2 (Xian et al., 10 Jun 2026). Probing identifies positions 12 and 18, and channels $\rho_{\text{cov}}$3, as effective locations for global appearance control. The token-to-map interface lifts token channels from 12 to 64, maps to a $\rho_{\text{cov}}$4 coarse map, applies a residual local aggregation branch, and progressively upsamples to $\rho_{\text{cov}}$5. With TiTok-32, the paper reports better fusion metrics than TiTok-64 and TiTok-128, and the two-slot STE setting gives the best M3FD ablation. The method reports trainable parameters of 1.3M, FLOPs of 304.5G, latency of 124.3 ms, memory of 2.78 GB, and strong benchmark results including M3FD EN 7.19, SD 47.35, SCD 1.85, SSIM 1.49 (Xian et al., 10 Jun 2026). The token space is therefore used as a low-dimensional non-spatial control plane for global coherence, while the 2D pathway preserves local detail.

5. Language-model computation and intention-level aggregation

Mixture of Tokens (MoT) defines Shared Aggregation Tokens as per-expert mixed representations created by aggregating tokens across different examples. For each token group, typically all examples at the same sequence position, the controller computes expert-specific weights over tokens by a softmax normalized across tokens rather than across experts, producing weights $\rho_{\text{cov}}$6 for token $\rho_{\text{cov}}$7 and expert $\rho_{\text{cov}}$8 (Antoniak et al., 2023). Each expert receives a single mixed token

$\rho_{\text{cov}}$9

processes it through its feed-forward network, and redistributes the output back to tokens using the same weights:

$\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$0

Because each expert processes exactly one mixed token per group, the load is balanced by design; there are no capacity constraints, token drops, or auxiliary load-balancing losses.

The method is explicitly designed to remain compatible with autoregressive training and generation by grouping tokens across sequences at the same position rather than within a sequence. In a small GPT-like setup on C4, the paper reports that MoT reaches the dense baseline’s final training loss in 24% of steps and 33% of wall-clock time, approximately a 3× increase in training speed over a vanilla Transformer. The reported configuration uses 4 transformer blocks, $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$1, $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$2 in the dense baseline, 512 experts, group size 32, context length 256, batch size 256, and 250k training steps (Antoniak et al., 2023). The paper also introduces transition tuning through the softmax temperature $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$3: as $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$4, the continuous MoT weights approach sparse expert-choice behavior, while higher $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$5 encourages mixing. The empirical claim is that low $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$6 during training hurts perplexity, even though deeper blocks tend to learn concentrated weights when $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$7 is learnable (Antoniak et al., 2023).

ARIA uses a different but related abstraction. It frames “Shared Aggregation Tokens” as intention clusters in a low-dimensional intention space $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$8, where semantically similar language actions share rewards rather than literal embeddings (Yang et al., 31 May 2025). Actions and observations are embedded with text-embedding-3-small and clustered by hierarchical agglomerative clustering; a history-action pair is then assigned an aggregated return $\sum_{j\in U}\alpha_{cj}^{(\ell)}=0$9 given by the average discounted reward over all examples with the same intention labels. This replaces high-variance token-sequence rewards with intention-conditioned shared rewards. The variance argument uses the law of total variance, and the paper’s theorem states that the gradient estimation error scales as $d$0 with the reduced variance $d$1 of the intention-conditioned estimator (Yang et al., 31 May 2025). Empirically, ARIA reports an average improvement of 9.95% across four downstream tasks, average win-rate gains of 9.67% in Bargaining and 9.83% in Negotiation, and average rewards of 35.25 in Twenty Questions and 36.00 in Guess My City. Here the “shared token” is best understood as a semantic macro-action or aggregation unit over many token sequences rather than a literal transformer token (Yang et al., 31 May 2025).

Taken together, these two papers show two distinct ways of sharing aggregation in language systems. MoT shares representations across examples inside a layer, while ARIA shares credit assignment across semantically clustered actions. One changes the forward operator; the other changes the reward estimator.

6. Distributed systems, token communication, and secure aggregation

In distributed function computation, token-based aggregation predates the recent deep-learning usage of the term. The TCM algorithm associates each node with a token carrying three fields, $d$2, and each node stores local memory $d$3 and a pointer $d$4 toward the latest outgoing edge of the highest-ID token (Salehkaleybar et al., 2017). Tokens coalesce when they meet, combining values through an associative rule $d$5, adding sizes, and preserving the larger UID. The distinguishing mechanism is chasing: lower-ID tokens that enter the event horizon of a higher-ID token follow stored path pointers instead of random walking. On complete graphs and Erdős–Rényi graphs with $d$6, the paper proves expected time $d$7, yielding a speedup of at least $d$8 over CRW; on 2D torus networks it proves expected time $d$9, with speedup at least $M$0 (Salehkaleybar et al., 2017). Simulation examples include $M$1 complete graphs, where TCM time is approximately 67 versus approximately 255 for CRW, and message complexity remains at least a constant factor lower.

Census likewise uses multiple circulating aggregation tokens in mobile ad hoc networks, but its emphasis is cover time under mobility. Each token is a portable aggregation state shared among nodes, with node-local variables visited, holder, and level governing biased random walks and temporary multi-hop gradients (Kulathumani et al., 2014). Unvisited nodes set up short-lived gradients that guide tokens toward unvisited regions. With $M$2 tokens, Census attains expected cover time $M$3 and message overhead $M$4, and simulations report that gradient bias is 12–14× faster than pure random walk while keeping exploration overhead approximately constant around 2 up to 4000 nodes (Kulathumani et al., 2014). The paper emphasizes scalability and robustness precisely because the token is structure-free and only lightly coordinated by local state.

An earlier token-based line formalizes tokens as transmission permits. In that model, each node has local variables status, value, and count; an active node may send its current aggregate to a random neighbor, after which the sender becomes inactive and the receiver becomes active by fusing the received state with its own (Saligrama et al., 2011). This supports Simple Random Walk, Coalescent Random Walk, Controlled Flooding, and hybrid CRW→CFLD schemes. The paper states that hybrid variants achieve per-node message complexity $M$5 and time complexity $M$6 on 2D mesh, torii, and random geometric graphs, with exact correctness guaranteed by associativity and the identity element of the aggregation operator (Saligrama et al., 2011). In this literature, the token is both a carrier of partial state and a scheduling primitive that limits message generation.

Token communication over outage channels introduces yet another use. SemPA-Look groups text tokens into packets so that the loss of any single packet preserves the semantics of the residual message as measured by Average Token Similarity, with cosine similarity computed from CLIP embeddings (Lee et al., 24 Jun 2025). The key surrogate is the Residual Semantic Score $M$7, which is justified by a low-$M$8 approximation of ATS. Rather than exhaustive partition search, SemPA-Look uses inter-packet candidate sampling with replacement and intra-packet lookahead sampling without replacement, leading to complexity $M$9 instead of exponential subset enumeration (Lee et al., 24 Jun 2025). On MS-COCO and WikiHow, it reports ATS and LPIPS close to exhaustive or GA baselines while reducing computation by up to 40× versus exhaustive search and about 10× versus GA. Here the shared aggregation object is the semantically designed packet rather than an attention token.

In information-theoretic secure aggregation, Shared Aggregation Tokens are uncoded groupwise shared keys. Each key $C_m\in\mathbb{R}^{(d/M)\times K}$0 is shared by a group $C_m\in\mathbb{R}^{(d/M)\times K}$1 of $C_m\in\mathbb{R}^{(d/M)\times K}$2 users and is independent of all other keys and model updates. Users split local models into $C_m\in\mathbb{R}^{(d/M)\times K}$3 blocks, mask each block with linear combinations of the keys in Round 1, and then send one linear combination of the aggregated masks in Round 2 so that the server can recover only the sum of surviving users’ models (Wan et al., 2022). The main result is parameter-dependent: if $C_m\in\mathbb{R}^{(d/M)\times K}$4, uncoded groupwise keys achieve the optimal per-user communication cost $C_m\in\mathbb{R}^{(d/M)\times K}$5 and $C_m\in\mathbb{R}^{(d/M)\times K}$6, matching the best coded-key schemes; if $C_m\in\mathbb{R}^{(d/M)\times K}$7, uncoded groupwise keys are strictly sub-optimal, and when $C_m\in\mathbb{R}^{(d/M)\times K}$8 the first-round rate must satisfy

$C_m\in\mathbb{R}^{(d/M)\times K}$9

An EC2 implementation reports model aggregation time reductions of 48%–53% versus SecAgg in the case $Vd$0, and 33%–44% in the case $Vd$1, $Vd$2 (Wan et al., 2022). In this setting, the shared token is cryptographic rather than representational, but the role is still aggregation through shared carriers with constrained cancellation.

7. Recurring design principles, trade-offs, and open issues

Several recurrent principles cut across these otherwise disparate literatures. First, shared aggregation tokens are often introduced to preserve a fixed interface: COMPASS keeps a modality-complete $Vd$3-slot input for the fusion head, ImAge keeps aggregation inside the transformer backbone rather than in a separate head, and MAG-VLAQ keeps aerial references anchored to a fixed shared prototype set even while allowing ground-side adaptation (Wang et al., 2 Apr 2026, Lu et al., 8 Nov 2025, Xu et al., 10 May 2026). This suggests that token sharing is frequently an interface-stabilization device as much as a representation device.

Second, sharing can deliver parameter efficiency or communication efficiency. ASG compresses embedding layers to 0.4%–0.5% of their original size while maintaining most task performance, MoT scales parameters without proportionally increasing runtime, TCM and Census reduce aggregation time or cover time through shared mobile carriers, and uncoded groupwise keys can match optimal secure-aggregation communication cost when $Vd$4 (V et al., 22 Sep 2025, Antoniak et al., 2023, Salehkaleybar et al., 2017, Kulathumani et al., 2014, Wan et al., 2022). These results suggest that a carefully designed shared carrier can replace a larger explicit mechanism.

Third, the same sharing can create inseparability. The strongest negative result is the prompt-injection impossibility theorem: when trusted instructions and untrusted data share embedding and value-aggregation pathways into control-relevant nodes, perfect Semantic-Faithful Control is impossible within that architecture (Pant et al., 25 Jun 2026). The contrast with the other domains is instructive. In secure aggregation, the shared carrier is useful only because cancellation structure is explicitly designed; in COMPASS, the replacement token works because alignment, shared-space regularization, and discriminative supervision are imposed; in MAG-VLAQ, query shifts are regularized by $Vd$5; in MoT, continuous mixing is controlled by temperature and group structure (Wan et al., 2022, Wang et al., 2 Apr 2026, Xu et al., 10 May 2026, Antoniak et al., 2023). A plausible implication is that sharing is benign when the interface or algebra is typed and constrained, and hazardous when it is unconstrained along a control-authoritative path.

Finally, every domain reports nontrivial trade-offs. In shared-embedding security, forbidding all untrusted contributions to control paths may degrade responsiveness; in ASG, too small $Vd$6 or $Vd$7 harms representational capacity; in ImAge, too many aggregation tokens interfere with patch-token context; in the 1D fusion interface, editing only one of positions 12 or 18 produces ghosting or blurring; in MoT, low temperature during training degrades perplexity; in SemPA-Look, packet size $Vd$8 must balance the packet-loss term against semantic loss; and in secure aggregation, uncoded keys lose optimality when group size is too small relative to tolerated dropouts (Pant et al., 25 Jun 2026, V et al., 22 Sep 2025, Lu et al., 8 Nov 2025, Xian et al., 10 Jun 2026, Antoniak et al., 2023, Lee et al., 24 Jun 2025, Wan et al., 2022). The literature therefore does not treat Shared Aggregation Tokens as a universally beneficial primitive. Rather, it treats them as a design pattern whose utility depends on what is shared, how aggregation is constrained, and whether the downstream pathway is descriptive, generative, control-authoritative, or adversarially exposed.