Unified Transformer Backbones & Modality Projections

• Unified Transformer Backbones with Modality-Specific Projections are architectures that use lightweight, modality-specific projections to convert diverse sensory inputs into a unified token space.
• They leverage a shared transformer core to fuse heterogeneous modalities, employing methods such as linear projections, VQ tokenizers, and cross-modal fusion techniques.
• Empirical results highlight improved parameter efficiency and versatility across tasks, showing strong performance compared to modality-specialized models.

Unified Transformer backbones with modality-specific projections constitute a central paradigm in multimodal foundation models. This architectural pattern leverages a single, highly parameter-shared transformer "core" to process tokens derived from heterogeneous modalities, with minor lightweight projections or modules responsible for mapping raw modality data into and out of the shared backbone. The strategy is mathematically grounded, empirically validated across domains, and exhibits strong parameter efficiency, transferability, and scalability.

1. Modality-Specific Projections: Formal Definitions and Implementations

Unified transformer architectures delegate initial modality adaptation almost exclusively to modality-specific projection layers or tokenizers, which standardize diverse input data into a common embedding space. The canonical workflow consists of:

• Patch or feature partitioning: Each input (image, DEM, surface normal map, albedo, language sequence, LiDAR point cloud, etc.) is divided into spatial or semantic patches, time windows, or wordpieces.
• Linear (or MLP) projection: For each modality $m$, a learnable projection $W^{(m)}_{\text{in}}$ is applied to map raw tokens $p_j^{(m)}$ (dimension varies by modality, e.g., pixels, channels) to the model’s feature dimension $d_\text{model}$:

$$x_j^{(m)} = W^{(m)}_{\text{in}} \cdot \mathrm{vec}(p_j^{(m)}) + b^{(m)}_{\text{in}}$$

• Modality specialization: Projections are parameterized separately per modality (e.g., $C_m$ channels differ; temporal, spatial, graph-based, or frequency-position encodings added as appropriate).
• Downstream mapping: At the output, further modality-specific heads (linear, MLP, or generative) may be employed for each required output domain.

Table 1 compares representative approaches:

Model	Projection Form	Modalities
Meta-Transformer (Zhang et al., 2023)	Linear patch/group for each $m$; maps to $D$	Text, images, point cloud, audio, ...
UBVMT (Ali et al., 2023)	Per-modality patch projector + axis embeddings	Face, ECG/PPG scalogram
MUTR (Yan et al., 2023)	Per-modality $W_m$, $b_m$ (for text/audio)	Language, audio, vision
The Moon's Many Faces (Sander et al., 8 May 2025)	Modality-specific input linear; VQ tokenizer	Grayscale, DEM, normals, albedo
UCFFormer (Yang et al., 2023)	Linear $W_m$ for each sensor	RGB, skeleton, inertial

This formal separation enables adaptation to diverse sensory data while keeping the core representation space unified and compatible for fusion in the transformer backbone.

2. Backbone Parameter Sharing and Modality Fusion

The central transformer backbone in all approaches is designed to be maximally shared across modalities. Uniform architectures (encoder-only, encoder-decoder, or causal autoregressive) consume the projected tokens with minimal awareness of originating modality. Key characteristics include:

• All-modal sharing: All self-attention and MLP/FFN layers (barring explicit exceptions) are shared for all token streams regardless of origin. Examples include the 6-encoder/6-decoder stack in (Sander et al., 8 May 2025), the 12-layer ViT backbone in Meta-Transformer (Zhang et al., 2023), and the 7B parameter AR transformer in Orthus (Kou et al., 2024).
• Cross-modal fusion: Mixed-modality sequences are handled natively, often via concatenation of tokens and global attention. When relevant, cross-attention (e.g., in decoders) conditions on context from all modalities.
• Position or axis encoding: Either shared or modality-specific; cross-modality alignment may be aided by spatial, temporal, axis-based, or learned positional embeddings.
• Architectural minimalism: In many cases (Meta-Transformer, UniTR, UBVMT), only $\sim$1-2% of parameters are in the modality-specific adapter, with the transformer backbone comprising 95+% of the model.

Flexible fusion arises from (a) the unified "token" standardization, and (b) the representational capacity of the transformer to model arbitrary intra-/inter-modal dependencies. Mechanisms such as automatic 2D/3D cross-modal set partitioning (UniTR (Wang et al., 2023)), shared global attention (UBVMT (Ali et al., 2023)), or fusion modules (adaptive fusion gate in BrainSymphony (Khajehnejad et al., 23 Jun 2025)) are used as needed.

3. Any-to-Any and Task-Agnostic Generalization

Unified transformer backbones with modality-specific projections support versatile any-to-any and task-agnostic translation paradigms. Notable strategies:

• Any-to-any masked autoencoding (Sander et al., 8 May 2025): Arbitrary subsets of modality tokens are masked and predicted, enabling the model to reconstruct any target modality from any available source set. This is operationalized via Dirichlet sampling to select context/target split, and a sum-of-cross-entropies loss across modalities.
• Unified shared/forked decoders (Hu et al., 2021, Li et al., 20 Jun 2025): Transformers are jointly pre-trained for multi-task objectives (classification, captioning, detection) across modalities. Task-specific output heads are attached after the shared representation.
• Modality-specific branching at depth: UniFork (Li et al., 20 Jun 2025) and Uni-X (Hao et al., 29 Sep 2025) demonstrate that optimal layer-wise modality alignment may require only partial sharing: initial/final blocks are modality- or task-specific, with sharing confined to semantically aligned middle layers.
• Plug-and-play adaptation (Zhang et al., 2024, Schwettmann et al., 2023): Any unimodal transformer can be recast as multimodal by learning lightweight "projection" adapters that inject tokens into the transformer, leaving all deep parameters unchanged and requiring no retraining.

Such flexibility supports not just unimodal tasks but also complex cross-modal mappings, translation, and compositional fusion.

4. Advanced Design Variants and Extensions

Several architectural innovations have emerged in this domain:

• Vector-quantized (VQ) tokenizers: Intermediate VQ codebooks enable discrete tokenization for modalities like DEM, normals, or albedo (cf. (Sander et al., 8 May 2025)). This enables discrete reconstruction losses, facilitates cross-modal mapping, and supports efficient token-space translation.
• Soft-VQ/continuous embeddings: Orthus (Kou et al., 2024) replaces hard VQ with softmax-based continuous codebook lookups, improving gradient flow and information conservation for image-to-language and image generation tasks.
• Mixture-of-Modality-Experts (MoME) (Bao et al., 2021): Feed-forward blocks can be replaced by pools of modality-specific experts; a deterministic routing determines which expert FFN handles each token.
• Factorized time-modality self-attention (Yang et al., 2023): Scalable joint attention across time and modality axes enhances the fusion of temporally aligned or sensor-heterogeneous streams.
• Cross-modal re-parameterization (Zhang et al., 2024): Auxiliary weights from an irrelevant modality are linearly combined (with per-layer learned scale) with the target weights, enhancing universal sequence modeling via parameter fusion, with no runtime overhead.
• Fusion modules (Khajehnejad et al., 23 Jun 2025): Dedicated modules such as Perceiver bottlenecks or adaptive gating can mediate information routing across spatial, temporal, and graph-structured modalities.

Each design is motivated by empirical and theoretical observations regarding gradient conflict, alignment, and information preservation across the representation hierarchy.

5. Empirical Results and Comparative Analyses

Unified transformer backbones with modality-specific projections consistently match or exceed the performance of modality- or task-specialized baselines on a broad range of benchmarks.

• Lunar reconstruction: The unified any-to-any backbone in (Sander et al., 8 May 2025) learned consistent, physically plausible mappings between lunar grayscale, DEM, normals, and albedo modalities without auxiliary supervision.
• Human action recognition: UCFFormer (Yang et al., 2023) achieved 99.04% accuracy with full sensor fusion and only $\sim$14M parameters, outperforming both unimodal and non-factorized attention models.
• Emotion recognition: UBVMT (Ali et al., 2023) demonstrated that as little as $\sim$1.2M parameters devoted to modality-specific projections sufficed for high performance on multimodal arousal/valence tasks across biosensor and visual modalities.
• Parameter sharing trade-offs: UniFork (Li et al., 20 Jun 2025) and Uni-X (Hao et al., 29 Sep 2025) provide quantitative ablations showing fully shared backbones compromise alignment and accuracy, while mid-depth branching/forking recovers or surpasses expert model performance. Uni-X (3B) achieved GenEval 82 in image generation, matching Bagel-14B+Diffusion, with much higher efficiency.
• Zero-shot and few-shot transfer: Meta-Transformer (Zhang et al., 2023) attains competitive results across 12 modalities using only frozen backbone weights and per-modality linear adapters.
• Efficiency and scaling: Approaches such as Cross-Modal Re-parameterization (Zhang et al., 2024) upgrade pretrained transformers into efficient multimodal models yielding consistent improvements (ImageNet, ShapeNetPart, Kinetics-400) with no additional inference cost.

These results underscore the universality of token-based sequence modeling learned by transformers and the sufficiency of minimal modality-specific adaptation for high-level multimodal intelligence.

6. Limitations, Open Challenges, and Theoretical Implications

Despite empirical success, several open questions and limitations persist:

• Gradient conflict: Full parameter sharing in AR transformers often induces destructive interference in shallow and deep layers, primarily due to mismatched low-level feature statistics; partial separation (Uni-X (Hao et al., 29 Sep 2025)) or branching (UniFork (Li et al., 20 Jun 2025)) can alleviate, but not entirely remove, residual conflict.
• Task interference: A fully shared backbone may impair late-stage feature recovery for generation (UniFork (Li et al., 20 Jun 2025)), necessitating task-specific heads or specialized late blocks.
• Projection bottlenecks: Shallow adapters may be insufficient for highly heterogeneous modalities or when fine-grained alignment is required (e.g., high-res video, irregular graphs).
• Modality scaling: As more domains (e.g., tabular, medical, graph) are added, the choice between fully frozen backbones (Meta-Transformer (Zhang et al., 2023)) and co-trained/fusion-adapted models depends on computational and statistical trade-offs.
• Semantic alignment depth: In cases where a frozen transformer is augmented via input projections, semantic translation from, e.g., image to language occurs mostly in the deep backbone layers, not in the projection (cf. "multimodal neurons" in (Schwettmann et al., 2023)). This suggests that universal transformers can support cross-modal reasoning when provided with suitable per-modality adapters.

A plausible implication is that the transformer backbone's universality arises not from explicit cross-modal design, but from its ability to support generic abstraction, so long as shallow adapters can present appropriately formatted input tokens.

7. Future Directions and Generalization Beyond Current Modalities

Future research is likely to focus on:

• Scaling up to hundreds of modalities: Integration of medical, scientific, time-series, event, molecular, and document modalities via increasingly generalized tokenizers.
• Automated or learned tokenization: Optimizing projection schemes for each modality, possibly including nonlinear or attention-based adapters.
• Dynamic parameter sharing: Routing, expert mixture, or soft-sharing architectures that allow flexible allocation of shared and modality-specific capacity per layer and per sample.
• Unified objectives: Extending any-to-any translation frameworks and multi-task heads to encompass complex, multi-step, or interaction-based outputs.
• Causality and interpretability: Identification and causal analysis of "multimodal neurons" (cf. (Schwettmann et al., 2023)) that mediate cross-domain representation fusion.
• Integration with retrieval and memory: Joint modeling of retrieval-augmented, event-driven, or external-memory–based fusion in the unified transformer context.

These developments will further clarify the theoretical and practical boundaries of unified transformer backbones equipped with modality-specific projections as the dominant architecture for multimodal foundation models.