Any-to-Any Multimodal Models

• Any-to-any multimodal models are unified architectures that process arbitrary combinations of input and output modalities (e.g., text, images, audio, video) through shared tokenization and fusion techniques.
• They employ techniques like discrete tokenization, transformer fusion, and modality-adaptive inference to ensure robust performance even with missing modalities.
• These models enable versatile applications from AI assistants to computational pathology by achieving scalable, cross-modal generation and understanding.

Any-to-any multimodal models are architectures explicitly designed to process arbitrary combinations of input and output modalities—such as text, images, audio, video, genomics, and scientific data—within a single unified system. Unlike classical unimodal or even text-to-X multimodal models, any-to-any systems support flexible, permutation-invariant mappings between multiple input and output modalities, facilitating tasks such as cross-modal generation, multi-modal understanding, and modality-conditioned reasoning. This paradigm addresses key challenges in heterogeneity, missing-modality robustness, computational scaling, and downstream generalizability, and spans diverse application domains from open-ended AI assistants to computational pathology.

1. Core Architectural Principles and Paradigms

Any-to-any models are characterized by several foundational architectural principles:

• Unified Modality Abstraction: All supported modalities are mapped into a shared representational or token space, enabling joint modeling, fusion, and cross-modal reasoning using a single model backbone (Li et al., 2024, Luo et al., 15 Oct 2025, Zhan et al., 2024, Tang et al., 2023).
• Discrete Tokenization and/or Unified Embedding: For efficient cross-modal autoregression or flow-based modeling, modalities such as speech, images, video, music, and structured data are discretized via modality-adapted quantizers (e.g., VQ-VAE, semantic codebooks), or aligned into shared continuous embedding spaces (Zhan et al., 2024, Cheng et al., 25 Jan 2026, Sun et al., 19 May 2025, Luo et al., 15 Oct 2025).
• Universal Fusion and Decoding: Architectures implement transformer-based fusion or flow-matching mechanisms capable of fusing arbitrary subsets of modalities through shared attention, mixture-of-experts adapters, or continuous-flow/rectified-flow modeling that eliminate the need for explicit inter-modality mapping heads (Sun et al., 19 May 2025, Li et al., 2024, Luo et al., 15 Oct 2025, Cheng et al., 25 Jan 2026).
• Modality-Adaptive Inference and Training: Any subset of modalities may be present at either training or inference time, with architectures supporting natural skipping of missing modalities (e.g., via token masking, attention over present tokens only, or parallel flow updates), removing the need for imputation or retraining (Sun et al., 19 May 2025, Wu et al., 2023, Zhan et al., 2024).
• Generic Loss Functions and Objectives: Systems generally optimize unified next-token prediction, flow-matching, or diffusion-based denoising objectives augmented by cross-modal or triplet alignment and reconstruction losses. These objectives guarantee flexible transfer and robust representation, even under partial modality availability (Luo et al., 15 Oct 2025, Sun et al., 19 May 2025, Li et al., 2024, Tang et al., 2023).

2. Representative Model Architectures

Multiple distinct any-to-any system designs have emerged, exemplified by the following models:

Model	Backbone	Modalities	Decoding Paradigm
NExT-OMNI	Discrete Flow (DFM) Transf.	Text/Image/Video/Aud	Parallel flow-matching
AnyGPT	Autoregressive Transformer	Text/Image/Speech/Mus	Discrete token AR
AR-Omni	Autoregressive Transformer	Text/Image/Speech	Single-token AR
Spider	Encoder-LLM-Decoder (+Ctrl)	Text/Img/Aud/Video	Prompted/Controller
CoDi/CoDi-2	Transformer+Diffusion	Text/Image/Audio	Latent diffusion
ALTER	Triplet-Transformer Fusion	WSI/Genomics/Report	Modal-adaptive pretrn.
StitchFusion	Multi-Adapter ViTs	Visual (arbitrary)	Weave-in-encoding

A representative case, NExT-OMNI (Luo et al., 15 Oct 2025), replaces all AR or diffusion token-generation heads with a parallel discrete flow-matching block, achieving bidirectional, unified cross-modal generation among text, images, audio, and video with a single Transformer backbone and lightweight modality-specific heads. In computational pathology, ALTER (Sun et al., 19 May 2025) fuses WSIs, genomics, and reports via a universal sequence transformer and modality-specific MoE adapters, yielding consistent improvements over unimodal and concatenative fusion baselines across survival prediction, subtyping, and generative tasks.

In Spider (Lai et al., 2024), a base X→X LLM system is augmented with an Efficient Decoders-Controller, enabling prompt-driven one-shot Any-to-Many generation, where multiple modalities are generated in a single turn. StitchFusion (Li et al., 2024) generalizes any-to-any modeling in the pure-vision domain using multi-scale adapters and parameter-efficient, weave-in-encoding fusion for arbitrary subsets of visual streams.

3. Pretraining Objectives and Modality Fusion

A universal feature of any-to-any models is the adaptation of loss functions and fusion strategies for multi-modal, permutation-invariant tasks:

• Intra-modal and Cross-modal Losses: Pretraining typically combines intra-modal masked modeling (e.g., masked language/region/pathway modeling), cross-modal contrastive (InfoNCE/CLIP-style) alignment, and higher-order objectives (triplet or joint consistency losses) (Sun et al., 19 May 2025, Li et al., 2024).
• Fusion Beyond Concatenation: Token or embedding sequences from all present modalities are fused either via shared Transformer attention blocks (universal/global self-attention), cross-attention (for late fusion or compositional generation), or flow-based multimodal coupling modules, with modality-specific "decoupling" experts restoring inductive statistical biases for each data type (Sun et al., 19 May 2025, Li et al., 2024, Li et al., 2024).
• Flexible Missing-Modality Handling: Fusion mechanisms are organized such that missing modalities are naturally skipped—e.g., absent tokens do not participate in self-attention or flow updates—allowing training strictly on partially-observed subsets and deployment on arbitrary combinations (Sun et al., 19 May 2025, Zhan et al., 2024, Tang et al., 2023).

ALTER, for instance, employs a hierarchical loss summing intra-modal MLM, cross-modal CLIP-style contrastive loss, and a global sample triplet loss that regularizes for class-level structure across fusion representations (Sun et al., 19 May 2025). StitchFusion's MultiAdapter propagates information across modality-specific ViT encoders at multiple scales via residual MLP adapters, with fusion occurring directly in the encoder pipeline (Li et al., 2024).

4. Application Domains and Quantitative Performance

Any-to-any multimodal models have been empirically validated in a spectrum of domains:

• Open-ended multimodal generation and understanding: Generalist architectures such as NExT-OMNI (Luo et al., 15 Oct 2025), AnyGPT (Zhan et al., 2024), and AR-Omni (Cheng et al., 25 Jan 2026) set state-of-the-art or competitive performance on image captioning, text-to-image, speech recognition, TTS, and audio/music understanding. For example, AR-Omni attains real-time streaming speech generation with a 0.88 real-time factor and 6.5% WER on VCTK zero-shot TTS (Cheng et al., 25 Jan 2026).
• Domain-specialist any-to-any: ALTER (Sun et al., 19 May 2025) in computational pathology demonstrates that triplet-fused, modality-adaptive modeling surpasses state-of-the-art on survival prediction (C-index +3.8% over SurvPath), subtyping (AUC +1.8% over ABMIL), mutation prediction, and WSI→report generation.
• Vision-exclusive any-to-any systems: StitchFusion matches or beats domain baselines across a range of segmentation setups, with only ~0.7 M additional parameters needed to fuse four visual modalities and achieving up to +11% mIoU improvements (Li et al., 2024).
• Benchmarks and ablations: Any-to-any models are evaluated on dedicated protocols such as ACON for cross-modal consistency (Chung et al., 30 May 2025), FysicsWorld for bidirectional four-modality understanding/generation/reasoning (Jiang et al., 14 Dec 2025), and custom protein-sequence/structure/text translation tasks (Chen et al., 2024).

5. Limitations and Consistency Challenges

Empirical analyses reveal that, while any-to-any models enjoy architectural and maintenance efficiency, their ability to achieve strong cross-modal consistency, invertibility, and equivariance is not guaranteed (Chung et al., 30 May 2025):

• Cyclic Consistency: Unified models do not always outperform paired specialist pipelines on pointwise cyclic consistency metrics; distributional alignment (equivariance) is weak but measurable only in models with strong semantic token alignment (e.g., VILA-U, Seed-X) (Chung et al., 30 May 2025).
• Shortcut learning and shallow fusion: FysicsWorld demonstrates that shallow concatenation or weak interdependence across modalities leads to performance deficits (10–20 points for speech+vision fusion) and failure modes on fusion-dependent reasoning (Jiang et al., 14 Dec 2025).
• Limited bidirectional invertibility: Structured analyses indicate that only distributional latent "vector" editing is preserved in current any-to-any architectures; direct invertibility between modalities (e.g., image↔text) remains a challenging research direction (Chung et al., 30 May 2025).
• Scaling and representation limitations: Data scarcity for certain modality pairs, codebook saturation, and computational costs for massive fusion models (up to 3B–8B parameters) are ongoing challenges (Zhan et al., 2024, Luo et al., 15 Oct 2025, Team, 5 Jan 2026).

6. Deployment, Scaling, and Architectural Trends

Serving and scaling any-to-any models introduces new system-level considerations:

• Computation-Path Heterogeneity: As shown in Cornserve (Ma et al., 16 Dec 2025), request heterogeneity (arbitrary input-output modality combinations) and executor heterogeneity (specialized encoders, decoders, AR Transformer cores) necessitate runtime planning and dynamic resource allocation, yielding up to 3.8× throughput and nearly 6× tail-latency improvements over monolithic serving (Ma et al., 16 Dec 2025).
• Plug-and-play Modularity: Several architectures support modular addition of new modalities without retraining core weights, using adapter-based plug-ins or explicitly compositional symbolic workflow representations. The latter, as in the symbolic task compiler (Chen et al., 24 Apr 2025), enables arbitrary task/program decomposition, editability, and composability across any modular pipeline.
• Scaling to tens of modalities: 4M-21 (Bachmann et al., 2024) demonstrates scaling to twenty-one continuous and discrete vision-related modalities using discrete tokenization, supporting new generation types (palette→depth, metadata→RGB) not previously modeled, without loss of overall unimodal and multimodal accuracy relative to prior (3–7 modality) baselines.

7. Future Directions and Open Challenges

Current research and benchmarks delineate several future directions for any-to-any modeling:

• Explicit semantic alignment and invertibility: Stronger semantic alignment (contrastive or cross-attention losses), cycle/equivariance regularization, and jointly learned tokenizers may increase pointwise cross-modal coherence (Chung et al., 30 May 2025, Jiang et al., 14 Dec 2025).
• Full-modality reasoning and multimodal memory: Benchmarks like FysicsWorld facilitate systematic progress on any-to-any multimodal reasoning, fusion-dependent tasks, and multi-turn dialogue, emphasizing the need for world-modeling and causal inference informed by domain structure (Jiang et al., 14 Dec 2025).
• Automatic workflow induction and agentic composition: Symbolic flow-based and agentic frameworks for task composition promise fine-grained editability, interruptibility, and programmatic integration of new modalities, surpassing monolithic neural systems in extensibility (Chen et al., 24 Apr 2025).
• Real-time, multi-modal and multi-lingual deployment: Achieving real-time, robust any-to-any operation across languages and streaming data is now becoming practical at the 7–8B parameter scale (e.g., HyperCLOVA X OMNI) (Team, 5 Jan 2026).

In sum, any-to-any multimodal modeling establishes a rigorous architectural, algorithmic, and systems foundation for next-generation generalist AI, encompassing not only universal cross-modal generation and understanding but also practical deployment, robustness to missing modalities, and fine-grained, extensible plug-in capacity. These capabilities are now core to both scientific and real-world AI systems (Sun et al., 19 May 2025, Luo et al., 15 Oct 2025, Zhan et al., 2024, Ma et al., 16 Dec 2025, Li et al., 2024, Lai et al., 2024, Tang et al., 2023, Chen et al., 2024).