Any2Any: Unified Arbitrary Modality Translation

Updated 3 July 2026

Any2Any is a framework enabling arbitrary mapping between diverse modalities, classes, or agents using a unified latent space and banked expert parameters.
It reparameterizes translation tasks to address combinatorial explosion and supports zero-shot adaptability even with missing or unseen modalities.
Real-world applications span remote sensing, motion generation, and collaborative perception, with empirical benchmarks demonstrating superior performance over traditional pairwise methods.

Any2Any refers broadly to algorithmic, architectural, or mathematical frameworks that allow arbitrary mapping or translation between multiple modalities, classes, agents, or entities in a system. Unlike pairwise or "one-to-one" formulations, Any2Any necessitates a unifying machinery that can generalize across all possible input–output combinations, supporting arbitrary source–target pairs (or sets), and in some cases, their intersectional fusion or imputation. Across disciplines, this paradigm appears in domains as diverse as remote sensing, machine learning for generative modeling, robotics, collaborative perception, quantum information, security, and wireless communication. The following sections survey leading methodologies and key results underpinning the modern theory and practice of Any2Any systems.

1. Foundational Formulations and Motivations

Any2Any is motivated by two ubiquitous bottlenecks: (a) combinatorial explosion as the number of modalities, classes, or embodied forms increases, and (b) the need for robust generalization and zero-shot adaptability in real-world, open-world, or incomplete-data scenarios. Canonical pairwise approaches scale as $O(N^2)$ in the number of entities, leading to prohibitive storage and supervision demands. Any2Any reparameterizes the mapping task as inference over shared latent spaces, banked expert parameters, or symbolic intermediates, and augments traditional training setups with architectural or algorithmic inductive biases that promote modality-agnostic (or target-agnostic) generalization.

In remote sensing, arbitrary modality translation is recast as diffusion over a geometrically-aligned latent manifold, meaning each modality represents a partial observation of an underlying semantic scene state, all linked via spatially coherent encoders and lightweight residual adapters (Chen et al., 4 Mar 2026). In motion generation, Any2Any conditioning enables the synthesis of 3D human motion from any subset of text, audio, trajectory, and other embeddings through a scalable masked modeling transformer (Li et al., 28 May 2026). In collaborative perception, universal feature translators instantiate parameter sets dynamically as functions of the source–target modality code, surpassing static adapters in both scalability and zero-shot generalization (Li et al., 18 May 2026).

2. Core Methodologies and Model Architectures

Latent-Space Anchoring and Unified Diffusion Backbones

A common principle across Any2Any frameworks is the decoupling of modality-specific projection from modality-agnostic semantic mapping, typically via a shared latent space, over which a unified backbone (often a Transformer-based diffusion network) operates. In remote sensing Any2Any (Chen et al., 4 Mar 2026), pretrained modality-specific VAEs encode each observation $x_k$ to a latent $z_k$ ; a single diffusion model $f_\theta$ then regresses between $z_i$ and $z_j$ using a direct $x_0$ prediction objective: $L_{z_0} = \mathbb{E}_{(x_i,x_j),t,\epsilon} \left\| z_j - f_\theta([z_t;z_i], c) \right\|_2^2$ where $c$ encodes timestep and one-hot source/target identifiers.

Lightweight residual adapters $R_j$ are appended to correct systematic latent mismatches: $x_k$ 0 Each $x_k$ 1 is trained with last-layer zero initialization and a stop-gradient calibration loss.

Modality-Intrinsic Code Spaces and Dynamic Translator Synthesis

Collaborative Any2Any perception (Li et al., 18 May 2026) eschews pairwise adapter training by embedding features into a modality-intrinsic latent space via an encoder $x_k$ 2. The codes $x_k$ 3 serve as semantically meaningful anchors; for each source–target pair, a router module computes combination coefficients $x_k$ 4 over a bank of expert parameters, dynamically synthesizing the required translation weights: $x_k$ 5 Where $x_k$ 6 computes a mapping descriptor (distance or concatenation) and $x_k$ 7 is an MLP.

This construction supports true zero-shot any-to-any translation: new modalities unseen at train time can be mapped into embedded code space $x_k$ 8, allowing the router to interpolate among pre-learned bases for on-the-fly adaptation.

Masked Modeling, Parallel Objectives and Multimodal Conditioning

AnyMo (Li et al., 28 May 2026) leverages a residual FSQ tokenizer and a masked modeling transformer for arbitrary-modality conditional motion synthesis. Given tokens for observed modalities, masked tokens for missing data, and embeddings for available conditions, a cross-attentional Transformer predicts all V+1 quantization streams in parallel, with an objective of the form: $x_k$ 9 where $z_k$ 0 is the per-stream cross entropy at masked positions, and $z_k$ 1 is the reconstruction MSE. This design supports both single-modal and multi-modal conditioning—text, audio, trajectory, or any combination—and allows for fine-grained control during inference.

Symbolic and Agentic Representation

Symbolic Any2Any inference formalizes every generative mapping as a tuple $z_k$ 2—functions, parameters, topology—using a LLM to analyze natural language instructions, generate function graphs, and compose executable dataflows in a training-free pipeline (Chen et al., 24 Apr 2025). Model flexibility emerges from the decoupling of task description, function inventory, and wiring: each variant of a generative task (image $z_k$ 3video, text $z_k$ 4audio, etc.) is rendered as a symbolic workflow, which can be compiled and refined iteratively.

3. Benchmarking, Experimental Results, and Scalability

Quantitative Comparison: Remote Sensing Modality Translation

Empirical studies on the RST-1M dataset (Chen et al., 4 Mar 2026) show that Any2Any surpasses all previous state-of-the-art pairwise translation methods (e.g., Pix2Pix, BBDM, ControlNet) in PSNR, SSIM, and RMSE across all 14 translation tasks, with a unified backbone covering all source–target pairs. Notably, zero-shot transfer to unseen directions (e.g., SAR $z_k$ 5PAN) yields plausible, high-fidelity outputs despite no direct paired supervision.

Method	SAR→RGB (PSNR)	NIR→MS (PSNR)	RGB→PAN (PSNR)
Pix2Pix	11.84	—	10.76
BBDM	19.50	—	14.66
Any2Any-L	25.20	—	33.45

Motion Generation and Multimodal Conditioning

AnyMo on OmniHuMo (Li et al., 28 May 2026), using >3M clips, achieves a FID of 55.6 for text $z_k$ 6motion synthesis (3B parameter model); multi-modal conditioning further reduces FID (e.g., Text+Trajectory $z_k$ 7 41.43), with recall R@1 and diversity also improved. Multi-modal inputs consistently enhanced output quality, validating the hybrid conditioning architecture.

Collaborative Perception

In the OPV2V-H benchmark (Li et al., 18 May 2026), UniTrans outperforms NegoCollab and other SOTA cross-modal fusion architectures, especially in zero-shot settings. Efficiency gains are substantial: UniTrans instantiates a single expert per task (CPU 6.9 ms) versus 3× slower mixtures in classic MoE.

Model	[email protected]	[email protected]	CPU Time (ms)
UniTrans	0.716	0.605	6.9
NegoCollab	0.662	0.538	89

Data and Compute Efficiency

Any2Any cross-embodiment transfer in humanoid robotics (Yang et al., 22 May 2026) demonstrates that with only 1% of the data and computation of from-scratch training, final tracking rewards are matched or surpassed, and convergence is achieved an order of magnitude faster. Low-rank adaptation (LoRA) applied selectively to dynamics-critical modules enables rapid generalization to new morphologies.

4. Robustness, Zero-Shot Generalization, and System Properties

Any2Any frameworks universally emphasize robustness to missing modalities, label permutations, or unseen conditions:

Remote Sensing: Adapters specialize decoders to each target, enabling direct translation between arbitrary subset pairs; fusion of latent anchors leverages all available supervision.
Collaborative Perception: As new modalities appear post-deployment, router modules in UniTrans can synthesize plausible translation parameters without retraining.
Motion Synthesis: Any subset of available modalities can condition the output; missing channels simply mask corresponding embeddings.
Symbolic Generation: Tasks with previously unencountered input–output topologies can be solved in a training-free, iterative-refinement regime.

Zero-shot results demonstrate model extensibility. For held-out translation pairs or unseen modalities, output quality degrades only modestly and remains competitive with direct-pair models (Chen et al., 4 Mar 2026, Li et al., 18 May 2026).

5. Limitations, Open Problems, and Future Research

Despite strong empirical and theoretical results, Any2Any frameworks have important limitations:

Dependence on Large, Aligned Datasets: Most latent backbone approaches require large, sparsely connected multi-modal datasets (e.g., RST-1M, OmniHuMo). Performance under limited or unaligned data remains an open problem (Chen et al., 4 Mar 2026, Li et al., 28 May 2026).
Frozen Encoder Bottlenecks: Freezing encoders and decoders (to anchor latent space) fixes their projection quality; domain drift or out-of-distribution modalities may degrade performance.
Computational Burden: Large VAE footprints and diffusion sampling cost (e.g., 250 steps per sample) constrain deployment for some applications. Work on accelerated samplers and efficient quantization is ongoing.
Extending Beyond Canonical Modalities: Current methods have not fully addressed text $z_k$ 8vision translation in remote sensing, or support seamless integration with textual or agentic instructions (Chen et al., 4 Mar 2026).
Structural Planning in Interleaved Tasks: Any2Any interleaved multimodal benchmarks (e.g. UniM (Li et al., 5 Mar 2026)) show that structure adherence (modality order, count) is still a bottleneck. Methods for dynamic, cognitive-style planning and cross-modal synergy remain active research frontiers.

Future avenues include leveraging unpaired or weakly supervised anchor losses, cycle-consistency in latent space, dynamic reward optimization, and full integration of symbolic planning with neural translation. Unified frameworks targeting temporal sequences, spatiotemporal fields, or high-dimensional multi-agent systems highlight the continued importance of general-purpose, scalable Any2Any architectures.

Key References:

"Any2Any: Unified Arbitrary Modality Translation for Remote Sensing" (Chen et al., 4 Mar 2026)
"AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling" (Li et al., 28 May 2026)
"One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception" (Li et al., 18 May 2026)
"Any2Any: Efficient Cross-Embodiment Transfer for Humanoid Whole-Body Tracking" (Yang et al., 22 May 2026)
"Symbolic Representation for Any-to-Any Generative Tasks" (Chen et al., 24 Apr 2025)
"UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark" (Li et al., 5 Mar 2026)
"Any2Any 3D Diffusion Models with Knowledge Transfer" (Wang et al., 10 May 2026)