Unified Token-Based Architectures

• Unified Token-Based Architectures are defined by representing diverse modalities as sequences of discrete tokens, enabling consistent processing across domains.
• They employ advanced token mixing, compression, and fusion mechanisms to optimize resource utilization and improve performance in tasks such as vision, language, and molecular science.
• These architectures support modular, multi-modal learning and universal approximation with practical applications in communication, authentication, and biological modeling.

Unified token-based architectures are a class of computational frameworks in which all modalities, tasks, or operations are represented and processed as sequences of discrete tokens. These architectures unify model design principles across domains—including vision, language, audio, molecular science, communications, and authentication—by utilizing token-centric representations to facilitate mixing, compression, inference, and transmission. Tokens serve as atomic, context-aware information units, enabling consistent implementation of core mechanisms such as next-token prediction, autoregressive decoding, and cross-modal fusion. Unified token-based design thus provides structural compatibility, operational flexibility, and optimal resource utilization across diverse modalities and tasks.

1. Principles of Unified Token Representations

Unified token-based architectures are defined by the use of tokens as the universal representation for heterogeneous data—text, visual patches, molecular graphs, authentication credentials, and more. Architectures such as Active Token Mixer (ATM) (Wei et al., 2022) and UniMoT (Guo et al., 1 Aug 2024) use specialized tokenizers (e.g., vector quantization-driven, causal masking modules) to transform complex inputs into sequences of discrete tokens that encapsulate high-level semantics. This enables models to read, generate, and reason about non-textual modalities identically to natural language, bridging modality-specific gaps and permitting compositional operations such as mixing, generation, and retrieval. The token-centric representation allows unified metrics and loss functions, e.g., cross-entropy over token sequences or mutual information objectives, which express task outputs as autoregressive token prediction problems (see Prot2Token (Pourmirzaei et al., 26 May 2025)).

2. Token Mixing, Fusion, and Compression Mechanisms

Token mixing and compression are generalized through matrix transformations, attention-based mixing, and hybrid merging/pruning operations. For instance, the Active Token Mixer (Wei et al., 2022) reformulates mixing from CNNs, Transformers, and MLPs into the unified expression

$$f(\mathbf{X})|_{\mathbf{x}_q} = \sum_{k \in \mathcal{N}(\mathbf{x}_q)} \omega^{(k \rightarrow q)} \cdot g(\mathbf{x}_k)$$

and augments this mechanics with adaptive, channel-wise contextual selection, allowing global receptive fields and efficient fusion. Token Transforming (Zeng et al., 6 Jun 2025) encapsulates pruning and merging into a general many-to-many matrix mapping,

$\mathbf{Y} = W \cdot \mathbf{X}$

where weights are assigned via similarity or informativeness, sidestepping token exclusivity and information loss. Similarly, MergeVQ (Li et al., 1 Apr 2025) merges tokens post-attention to decouple coarse semantics from fine-grained details, enabling both representation learning and generative reconstruction. Hybrid reduction schemes for state-space models (Zhan et al., 16 Oct 2024) combine importance metrics and cosine similarity to balance redundancy removal and information preservation, defining intra-layer reduction policies.

3. Unified Multi-modal and Cross-domain Architectures

Unified token frameworks extend to multi-modal and cross-domain applications by serializing different modalities into token sequences and fusing them in shared embedding or processing spaces. MMTrack (Zheng et al., 2023) converts language descriptions and vision bounding boxes into a unified token sequence for vision-language tracking. UniMoT (Guo et al., 1 Aug 2024) uses vector quantization to expand LLM vocabularies with molecule tokens, unifying molecule-text processing for comprehension and generation. Communication frameworks such as TokCom (Qiao et al., 17 Feb 2025) and UniToCom (Wei et al., 2 Jul 2025) apply transformer-based tokenization and next-token prediction for multimodal semantic transmission, integrating GenIB (generative information bottleneck) principles to minimize mutual information subject to representational fidelity:

$$\min_{p_{T|X}} I(X; T) \quad \text{subject to} \quad I(\tilde{T}; X) \geq \chi$$

resulting in scalable architectures for wireless communications and context-aware semantic error correction.

4. Modularization and Task-Specific Token Optimization

Within unified frameworks, modularization is achieved by role-specific or task-specific token optimization. RoleRAG (Zhu et al., 21 May 2025) showcases a single, frozen LLM executing distinct retrieval-augmented generation modules via tuned role tokens (e.g., [QUERY_GRAPH], [JUDGE], etc.), with only the embeddings of these tokens being trainable. This design enables efficient multi-tasking, flexible extension, and significant resource reduction. Prot2Token (Pourmirzaei et al., 26 May 2025) employs learnable task tokens to steer an autoregressive decoder across a variety of protein modeling tasks, generalizing all outputs to token sequences and facilitating pseudo-universal next-token prediction via

$$p(x) = \prod_{t=1}^T p_\theta(x_t | x_1, \ldots, x_{t-1})$$

Such architectures streamline deployment, multi-task learning, and cross-task regularization.

5. Theoretical Foundations and Universal Approximation

Unified token-based design receives rigorous theoretical grounding in recent works (Cheng et al., 30 Jun 2025) that define sufficient conditions for universal approximation property (UAP) in transformer-type architectures. The framework splits block composition into token mixing (attention-like) and token-wise feedforward operations and identifies token distinguishability as critical for UAP. Token mixing must map distinct tokens to distinct representations, formalized via analyticity conditions on the attention kernel. The universal expressivity of token-based architectures is shown to hold for transformers with softmax, RBF, kernel-based, sparse, and convolutional (symmetry-preserving) attention, provided feedforward layers are sufficiently nonlinear and the attention mechanism distinguishes tokens within parameter families. This framework not only generalizes prior results but guides engineering of architectures with selected symmetries (cyclic, dihedral) and controlled connectivity.

6. Efficiency, Scalability, and Practical Impact

Unified token architectures offer substantial gains in efficiency, scalability, and domain transferability. Empirical evidence has demonstrated the following:

• Vision models (ATMNet (Wei et al., 2022), MergeVQ (Li et al., 1 Apr 2025)) achieve SOTA top-1 accuracy on ImageNet with reduced FLOPs and parameter counts.
• Compression frameworks (Token Transforming (Zeng et al., 6 Jun 2025), SSM token reduction (Zhan et al., 16 Oct 2024)) deliver acceleration (up to 1.5× speedup; 34–43% FLOPs reduction) with marginal or negligible accuracy drops, confirmed across classification, segmentation, detection, and multimodal LLM tasks.
• Large-scale, modular retrieval and generation systems (RoleRAG (Zhu et al., 21 May 2025)) achieve notable improvements (EM score increases of 16–64%) over previous RAG methods, while maintaining parameter efficiency.
• In protein modeling (Prot2Token (Pourmirzaei et al., 26 May 2025)), unified decoding yields up to ~1000× speedup over specialized alternatives (AlphaFold2-MSA) and competitive accuracy across task types.

Scalability extends to authentication (TrustZero (Dumitrescu et al., 14 Feb 2025), CMS SI (Yzquierdo et al., 23 May 2024)) where token-based models replace legacy identity-proxy paradigms, leverage cryptographic attestation chains, and fit seamlessly into distributed infrastructure, supporting both capability-based resource control and cross-organizational trust.

7. Future Directions and Challenges

Current research identifies multiple frontiers for unified token-based architectures:

• Extending adaptive token mixing for temporal and multimodal domains (ATMNet, future multimodal trackers)
• Designing universal tokenizers that operate consistently across modalities while maintaining joint embedding spaces for multimodal models (TokCom, UniToCom)
• Integrating computational collaboration (on-device, edge, cloud) to manage large generative model complexity (TokCom, UniToCom)
• Ensuring privacy and robustness of token representations against adversarial and inference attacks, with attention to secure error correction and semantic leakage prevention
• Theoretical work to further quantify the relationships between connectivity, distinguishability, inductive bias (symmetries), and minimal feedforward depth required for universal approximation.

A plausible implication is that unified token-based paradigms will continue to drive architectural innovations across domains, from intelligent wireless networks to large-scale multimodal reasoning, authentication infrastructure, and biological modeling. The consistent abstraction and operational flexibility of the token-centric view position it as a meta-architecture for scalable, context-aware, and efficient deep learning systems.