Any-to-Any Training Strategy

• Any-to-any training strategy is a framework that maps arbitrary valid input–output pairs within structured domains, enabling flexible cross-modal transfers.
• It leverages modular architectures such as factorization through intermediate structures and shared latent spaces to address diverse objectives across reinforcement learning, perception, and voice conversion.
• Empirical studies show improved transfer efficiency and structural validity, though challenges remain in robustness, domain adaptation, and privacy safeguards.

Any-to-Any Training Strategy denotes a family of training regimes in which a model, or a compositional set of model components, is prepared to map arbitrary valid inputs to arbitrary valid outputs inside a structured domain, rather than only a fixed source–target pair. Across the literature, the phrase is used for transfer from pure-strategy opponents to arbitrary opponent mixtures in multi-agent reinforcement learning, for translation between arbitrary feature modalities in heterogeneous collaborative perception, for voice conversion between arbitrary speakers, for interleaved multimodal understanding and generation across text, image, audio, video, document, code, and 3D, and for transfer between humanoid embodiments (Smith et al., 2020, Li et al., 18 May 2026, Lin et al., 2020, Tu et al., 10 Oct 2025, Li et al., 5 Mar 2026, Tang et al., 2023, Wu et al., 2023, Li et al., 2024, Yeo et al., 15 Jun 2026, Cha et al., 17 Dec 2025, Luo et al., 15 Oct 2025, Yang et al., 22 May 2026). The common objective is not uniform architecture, but a shared operational property: a single training recipe should support a large family of source conditions and target conditions, often with zero-shot or parameter-efficient deployment to configurations not explicitly optimized as one-off tasks.

1. Domain-specific meanings of “any-to-any”

The term is domain-relative rather than universal. In Q-Mixing, the problem is a two-player Markov game in which the opponent has a finite set of pure strategies, and the goal is to construct a policy that performs well against any mixture $w \in \Delta^K$ using only Q-functions trained against pure opponents (Smith et al., 2020). In UniTrans, the objective is to realize mappings $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$ for arbitrary source–target modality pairs in collaborative perception, including unseen modalities at inference (Li et al., 18 May 2026). In FragmentVC, MediumVC, and O_O-VC, “any-to-any” means conversion from arbitrary source speakers to arbitrary target speakers, including unseen speakers and, in O_O-VC, new languages (Lin et al., 2020, Gu et al., 2021, Tu et al., 10 Oct 2025).

In multimodal foundation models, the term expands further. UniM defines any-to-any interleaved multimodal learning as accepting sequences that interleave placeholders for text, image, audio, video, document, code, and 3D, and producing outputs that may themselves be arbitrarily interleaved across the same modalities (Li et al., 5 Mar 2026). CoDi-2, NExT-GPT, OmniFlow, MUNI, FlowBind, and NExT-OMNI each operationalize this idea differently—through language-centric continuous conditioning, modality signal tokens, multi-modal rectified flows, routed shared latents, bidirectional ODE flows, or discrete flow matching—but all aim at arbitrary cross-modal input–output mappings under one training umbrella (Tang et al., 2023, Wu et al., 2023, Li et al., 2024, Yeo et al., 15 Jun 2026, Cha et al., 17 Dec 2025, Luo et al., 15 Oct 2025).

A plausible implication is that “any-to-any” is best understood as a closure property over a task family. The admissible family may be a simplex over known opponents, a set of known or unseen feature modalities, a speaker space, a subset lattice over modalities, or a morphology class of robots. The phrase therefore indicates breadth of transferable conditioning, not a single canonical loss or architecture.

2. Recurrent architectural patterns

A first recurrent pattern is factorization through reusable intermediate structure. Q-Mixing learns separate best-response Q-functions $Q^{(i)}$ against pure opponents and approximates the mixed-opponent value by

$$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$

thereby converting a combinatorial target family into a linear composition over trained components (Smith et al., 2020). UniTrans uses an analogous but parametric mechanism: a Translator Parameter Bank stores reusable expert parameters, and a mapping-specific translator is instantiated by

$$\Theta_{s\to t} = \theta^{(0)} + \sum_{k=1}^{K} \alpha_k \theta^{(k)},$$

with the coefficients predicted from a modality mapping descriptor in a modality-intrinsic latent space (Li et al., 18 May 2026).

A second pattern is the construction of shared latent spaces that mediate arbitrary conditional routes. FlowBind defines a learnable shared latent $z^* = H_\phi(z_S)$ and modality-specific invertible ODE flows that can be integrated backward from a source modality into $z^*$ and forward into a target modality (Cha et al., 17 Dec 2025). MUNI replaces deterministic bridging with a shared stochastic latent, modality-specific decoders, and a learned flow-based prior; its routed objective is explicitly designed to make subset latents predictive for missing modalities while keeping the shared latent minimal (Yeo et al., 15 Jun 2026). UniTrans also belongs to this family, but uses a modality-intrinsic latent $Z$ built from channel statistics and Gram descriptors rather than a generative shared prior (Li et al., 18 May 2026).

A third pattern is modular alignment around a large pretrained core. NExT-GPT keeps ImageBind, Vicuna-7B, and the diffusion decoders frozen, and trains only an input projection, output projections, and LoRA adapters—131M trainable parameters, roughly 1% of the total system capacity (Wu et al., 2023). CoDi-2 similarly aligns modalities with language on both the encoding and generation sides, with the MLLM emitting continuous conditioning features that drive pretrained diffusion decoders (Tang et al., 2023). NExT-OMNI uses a single Qwen2.5-7B-based backbone and lightweight modality heads over unified discrete and continuous codebook embeddings, replacing decoupled pipelines with a bidirectionally fused encoder that supports understanding, generation, and retrieval in one representation space (Luo et al., 15 Oct 2025).

A fourth pattern is deliberate asymmetry. FragmentVC aligns source phonetic structure and target spectral structure through a two-stage curriculum, while MediumVC converts $X_i \rightarrow Y_i \rightarrow X_i$ through a specific-speaker intermediate, and O_O-VC trains on synthetic source–target pairs with identical linguistic content and aligned durations (Lin et al., 2020, Gu et al., 2021, Tu et al., 10 Oct 2025). These strategies reject symmetric autoencoding as the default any-to-any recipe and instead use asymmetric factorization to force content–speaker separation or direct source–target alignment.

3. Optimization regimes and objective design

Despite the conceptual similarity, the optimization layer differs sharply by domain. In reinforcement learning, Q-Mixing uses standard off-policy Q-learning, optionally with Double DQN, and can augment deployment with an opponent classifier trained in parallel via cross-entropy on replay-buffer data (Smith et al., 2020). In collaborative perception, UniTrans uses a two-stage objective: InfoNCE and modality classification for modality-intrinsic representation pretraining, followed by feature distillation, downstream task loss, routing consistency, and router regularization during translator expert learning (Li et al., 18 May 2026).

Voice-conversion methods are dominated by reconstruction losses, but the training semantics differ. FragmentVC trains end-to-end with reconstruction loss only,

$L_{\text{rec}} = \|\hat{S}-S\|_1,$

using a frozen Wav2Vec 2.0 encoder, a target spectrogram bank, and a two-stage curriculum that decays the probability that the source utterance appears among target utterances from 1.0 to 0.0 between steps 50k and 150k (Lin et al., 2020). MediumVC also uses L1 reconstruction, but places it inside a two-stage asymmetrical pipeline $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$0 built around synthetic specific-speaker speeches as intermedium features (Gu et al., 2021). O_O-VC adds a CVAE+flow+GAN stack with mel L1, KL divergence, adversarial, and feature-matching terms, and splits training into synthetic conversion followed by transcript-free real reconstruction fine-tuning (Tu et al., 10 Oct 2025).

Multimodal any-to-any systems exhibit the widest objective diversity. UniM and UniMA combine autoregressive cross-entropy over interleaved sequences with auxiliary structure-aware losses for modality-type coverage, placeholder counts, structural validity, and cross-modal alignment, while the agentic baseline additionally supervises intermediate reasoning artifacts such as TCDC, plan strings, tool lists, and final reports (Li et al., 5 Mar 2026). CoDi-2 combines text causal loss, feature regression to aligned encoder targets, and diffusion denoising loss that backpropagates through the MLLM’s continuous outputs (Tang et al., 2023). OmniFlow uses multi-modal rectified-flow regression on task-encoding paths through a noise-level cube, with modality-specific target velocities $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$1, plus a generalized multi-modal classifier-free guidance mechanism at inference (Li et al., 2024). FlowBind reduces training to a single flow-matching objective over straight paths between per-modality latents and a learnable shared anchor, whereas MUNI uses a routed multimodal variational objective with self-detached posteriors and prior gating to full and leave-one-out subsets (Cha et al., 17 Dec 2025, Yeo et al., 15 Jun 2026). NExT-OMNI trains discrete flow matching by expected cross-entropy against the posterior $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$2, augmented by image and audio reconstruction regularizers balanced by GradNorm (Luo et al., 15 Oct 2025).

Robotics introduces yet another regime. Any2Any first performs kinematic alignment between source and target humanoids and then optimizes only LoRA-based PEFT parameters in dynamics-sensitive modules under PPO, while freezing the source specialist backbone and preserving reward design and domain randomization from pretraining (Yang et al., 22 May 2026). This suggests that any-to-any training can be expressed not only as generalized conditional generation, but also as a structured restriction on which parameters may move during transfer.

4. Deployment, zero-shot behavior, and transfer mechanics

A central distinction is whether any-to-any capability is realized at inference without further optimization, or through a lightweight adaptation phase. Q-Mixing is explicitly zero-shot with respect to new opponent mixtures: once the pure-opponent Q-functions are learned, deployment to any $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$3 requires only computing the mixed value and acting greedily or via softmax, with no further RL training (Smith et al., 2020). UniTrans likewise instantiates translators on the fly for arbitrary source–target modality pairs by computing modality codes, a mapping descriptor, routing coefficients, and then a single parameter-combined translator, with no retraining or fine-tuning needed for new modality pairs (Li et al., 18 May 2026).

In multimodal generative systems, the dominant deployment pattern is conditional decoding over a unified token or latent stream. NExT-GPT routes LLM-emitted modality signal tokens through modality-specific output projections into frozen diffusion condition encoders, so the same model can answer in text or activate image, audio, and video generation on demand (Wu et al., 2023). CoDi-2 and OmniFlow perform synchronized multimodal generation by keeping modalities aligned in a shared conditioning space or along shared rectified-flow time coordinates, which allows intertwined outputs such as image–audio or video–audio to be generated in parallel (Tang et al., 2023, Li et al., 2024). FlowBind composes backward ODE integration from arbitrary observed modalities into a shared latent with forward integration into arbitrary target modalities, and multi-source conditioning is handled by averaging shared-latent estimates from each source (Cha et al., 17 Dec 2025). NExT-OMNI replaces autoregressive token-by-token decoding with parallel discrete flow updates over the full sequence, and its Dynamic Length Generation Strategy adjusts response length through EOS confidence while preserving blockwise parallel refinement (Luo et al., 15 Oct 2025).

Other systems are not zero-shot in this strict sense, but still qualify as any-to-any because they minimize target-side optimization. Any2Any transfers a pretrained whole-body tracking specialist to a new humanoid embodiment with kinematic alignment plus PEFT on selected dynamics-sensitive modules, using only a small amount of target data and compute (Yang et al., 22 May 2026). O_O-VC generalizes to unseen speakers and new languages after a two-phase training schedule, but still relies on a training-time synthetic-pair construction and a subsequent real-data fine-tuning phase (Tu et al., 10 Oct 2025). This suggests that “any-to-any” should not be conflated with “single frozen universal model”; in part of the literature it instead means universal transferability under sharply constrained adaptation.

5. Empirical record across domains

The empirical literature reports gains in coverage, structural validity, or transfer efficiency rather than a single universal benchmark.

Setting	Representative finding	Paper
Multi-agent RL	In grid-world soccer, Q-Mixing achieved broadly stronger coverage than BR(Uniform); in Gathering, BR$A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$4 achieved $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$5 vs $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$6 for BR$A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$7	(Smith et al., 2020)
Heterogeneous collaborative perception	On OPV2V-H, UniTrans reached $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$8 [email protected] / $A_{s\to t}: F_{m_s}\rightarrow F_{m_t}$9 [email protected]; on DAIR-V2X, $Q^{(i)}$0	(Li et al., 18 May 2026)
Any-to-any voice conversion	FragmentVC reported SV Accuracy $Q^{(i)}$1 on seen-to-seen and $Q^{(i)}$2 on unseen-to-unseen; naturalness MOS was $Q^{(i)}$3	(Lin et al., 2020)
Synthetic-pair voice conversion	O_O-VC reported a $Q^{(i)}$4 relative reduction in word error rate and a $Q^{(i)}$5 improvement in speaker cosine similarity	(Tu et al., 10 Oct 2025)
Interleaved multimodal benchmark	UniM contains 31,026 instances; UniMA reaches StS $Q^{(i)}$6–$Q^{(i)}$7, LeS $Q^{(i)}$8–$Q^{(i)}$9, SQCS $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$0 with $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$1, and ICS approaches $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$2	(Li et al., 5 Mar 2026)
Cross-embodiment humanoid transfer	Any2Any transferred Sonic from Unitree G1 to LimX Oli and LimX Luna using only $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$3 of the compute and data required for full training	(Yang et al., 22 May 2026)

Additional evidence comes from multimodal generation. NExT-GPT reports best zero-shot text-to-video FID $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$4 and CLIPSIM $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$5 on MSR-VTT, while remaining competitive on text-to-image and text-to-audio benchmarks (Wu et al., 2023). OmniFlow reports FID $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$6 and CLIP $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$7 on MSCOCO-30k for text-to-image, and FAD $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$8 with CLAP $$Q_{\text{mix}}(s,a;\mathbf{w}) \approx \sum_{i=1}^{K} w_i Q^{(i)}(s,a),$$9 on AudioCaps for text-to-audio (Li et al., 2024). FlowBind reports competitive quality while requiring up to $$\Theta_{s\to t} = \theta^{(0)} + \sum_{k=1}^{K} \alpha_k \theta^{(k)},$$0 fewer parameters and training $$\Theta_{s\to t} = \theta^{(0)} + \sum_{k=1}^{K} \alpha_k \theta^{(k)},$$1 faster than prior methods, and MUNI reports its largest gains on unconditional coherence rather than only conditional generation (Cha et al., 17 Dec 2025, Yeo et al., 15 Jun 2026). NExT-OMNI reports strong averages on OmniBench, WorldSense, and AV-Odyssey, leads OpenING multi-turn interaction among open unified models, and improves average Top-5 retrieval accuracy to $$\Theta_{s\to t} = \theta^{(0)} + \sum_{k=1}^{K} \alpha_k \theta^{(k)},$$2 on InfoSeek, OVEN, FashionIQ, and CIRR (Luo et al., 15 Oct 2025).

6. Limitations, controversies, and open directions

The main limitation is that “any” is always bounded. Q-Mixing generalizes to any mixture over a known set of pure opponents; mixtures involving unseen opponent types lie outside the convex hull of the training basis (Smith et al., 2020). UniTrans explicitly notes that if a new modality lies far outside the training latent manifold, mapping estimates may be unreliable (Li et al., 18 May 2026). FlowBind depends on strong frozen modality-specific encoders and decoders and does not optimize an explicit likelihood over its shared latent; this trades tractable joint modeling for efficiency (Cha et al., 17 Dec 2025). MUNI warns that if the prior-weighting hyperparameters are too large, unconditional coherence can dominate at the expense of conditional routes, while disabling self-detach causes modality-private details to leak into the shared latent (Yeo et al., 15 Jun 2026).

A second limitation is structural instability. Router collapse is a concern in UniTrans and motivates routing-consistency and load-balancing losses (Li et al., 18 May 2026). UniM documents common structural lapses such as incorrect placeholder counts, missing modalities, or incoherent interleavings, and makes these part of the benchmark itself through StS, LeS, and ICS (Li et al., 5 Mar 2026). NExT-OMNI identifies a granularity trade-off: without reconstruction regularizers, discrete flow matching can bias the shared representation toward semantics and degrade fine-grained generation, whereas too many sub-codebooks increase generation difficulty (Luo et al., 15 Oct 2025).

A third limitation is that unified training does not eliminate deployment bottlenecks inherited from downstream generators. CoDi-2 and NExT-GPT both depend on pretrained diffusion decoders, so overall fidelity and latency remain bounded by those modules (Tang et al., 2023, Wu et al., 2023). OmniFlow notes that some text-generation results still lag specialist captioners, in part because noisy weakly labeled training texts differ from benchmark caption distributions (Li et al., 2024). FragmentVC depends on vocoder quality, and O_O-VC depends on the fidelity and diversity of the multispeaker TTS used to synthesize aligned training pairs (Lin et al., 2020, Tu et al., 10 Oct 2025).

The literature also contains a substantive privacy controversy. In asynchronous voice anonymization, targeted training that maps all utterances to a designated speaker can improve unlinkability, but the designated identity is an actual speaker, creating privacy and impersonation risks. The any-to-any alternative replaces the fixed target with a pseudo-speaker defined as the batch mean of anonymized embeddings and optimizes a batch mean loss, thereby improving unlinkability without designating a real person as the anonymization sink (Chen et al., 21 Aug 2025). The same work shows that unlinkability remains fragile under adaptive perturbation-removal attacks and degrades out of domain, indicating that any-to-any objectives do not automatically confer adversarial robustness.

Future directions in the cited literature are consistent: non-linear or observation-aware mixing beyond linear composition, richer Bayesian or recurrent belief updates, stronger structure-aware decoding constraints, route-aware latent objectives, scaling to continuous or open-ended modality spaces, improved robustness to unseen domains and adaptive attacks, and increasingly explicit reuse of pretrained specialists rather than monolithic retraining (Smith et al., 2020, Li et al., 18 May 2026, Li et al., 5 Mar 2026, Yeo et al., 15 Jun 2026, Chen et al., 21 Aug 2025, Yang et al., 22 May 2026). This suggests that the mature form of any-to-any training is likely to remain modular: aligned latent spaces, reusable experts, structured routing, and targeted adaptation, rather than a single universally optimal backbone.