Cross-Modal Translation & Alignment

• Cross-Modal Translation and Alignment is a framework that maps information between different modalities and aligns them into a shared latent space while preserving semantic equivalence.
• It employs techniques like optimal transport, adversarial training, and cross-modal attention to ensure modality-specific features remain comparable.
• Applications span speech-text translation, affect recognition, retrieval, and biomedical analysis, demonstrating robust performance even with sparse or noisy paired data.

Cross-Modal Translation and Alignment (CMTA) denotes a family of methods that combine two operations across heterogeneous data types: a translation step, in which information from one modality is mapped into another modality or into a shared latent representation, and an alignment step, in which paired observations are constrained to become structurally compatible in that shared space. In the recent literature, this label spans zero-shot speech–text translation with multilingual fixed-size representations, multimodal affect recognition in which a stronger modality shapes a weaker one, cross-lingual cross-modal retrieval without human-annotated vision–target-language pairs, pathology–genomics survival modeling, sign language translation, zero-pair image translation, RGB–infrared detection, and linear-algebraic formulations of exact latent matching (Duquenne et al., 2022, Rajan et al., 2020, Wang et al., 2023, Zhou et al., 2023, Zhao et al., 2023, Wang et al., 2019, Yuan et al., 2022, Kamboj et al., 19 Mar 2025). Taken together, these systems suggest that CMTA is best understood as a research program centered on how to preserve semantic equivalence while respecting modality-specific structure.

1. Scope and defining characteristics

A common formulation recurs across otherwise distant tasks. In affect recognition, a stronger modality guides a weaker one by translating weaker features into stronger-modality features and aligning the resulting latent code with a stronger-modality latent through a correlation-based objective, while deployment uses only the weaker modality (Rajan et al., 2020). In speech translation, CMTA appears as sequence-level speech–text alignment, either through optimal transport and cross-modal mixup or through adversarial learning of a modality-invariant space shared by speech and text (Zhou et al., 2023, Zhang et al., 2023). In pathology–genomics survival analysis, CMTA is explicitly named as a framework that uses parallel encoder–decoder structures, cross-modal attention, and representation alignment to explore intrinsic cross-modal correlations while preserving complementary information (Zhou et al., 2023).

The same logic extends beyond supervised paired fusion. In cross-lingual cross-modal retrieval, CL2CM aligns source and target texts first and then distills that cross-lingual structure into vision–target-language alignment, thereby avoiding direct dependence on human-annotated vision–target-language pairs (Wang et al., 2023). In unpaired cross-lingual image captioning, Cross2StrA uses scene graphs as a semantic pivot for image–pivot-language alignment and constituency trees as a syntactic pivot for pivot–target-language alignment, then couples both stages with back-translation (Wu et al., 2023). In zero-pair image translation, mix-and-match networks align encoder bottlenecks so that an unseen encoder–decoder pair can be composed at test time even when no direct training pairs existed for that source–target modality pair (Wang et al., 2019).

This breadth matters conceptually. CMTA is not restricted to one pairing such as image–text or speech–text. The surveyed work covers speech, text, images, videos, pathology slides, genomic profiles, RGB–infrared imagery, and sign videos, with outputs ranging from retrieval scores to survival hazards and translated sentences (Yuan et al., 2022, Zhao et al., 2023, Hu et al., 10 Jul 2026). A plausible implication is that the unifying object is not the modality pair itself, but the requirement that semantically corresponding entities remain comparable after passing through modality-specific encoders.

2. Core technical primitives

The translation component is usually implemented as either reconstruction into another modality or explicit transport of local features. In SEW and CM-StEW, the weaker modality is encoded into a latent code and decoded into an approximation of the stronger modality; this latent is simultaneously used for downstream prediction, making it a weaker-modality-driven but multimodally constrained representation (Rajan et al., 2020, Rajan et al., 2021). In T-Modules, multilingual speech and text are encoded in a joint fixed-size representation space, and different decoders recover modality- and language-specific outputs without requiring cross-modal labeled translation data; the framework also reports the first zero-shot direct speech-to-speech and text-to-speech translation results (Duquenne et al., 2022). In pathology–genomics CMTA, each modality is translated into the other modality’s representational style through decoders, and the translated representations recalibrate the original intra-modal features (Zhou et al., 2023).

Alignment is instantiated by several mathematically distinct mechanisms. One family maximizes cross-view correlation. SEW uses a Deep CCA objective on weaker-driven and stronger-driven latent codes, while CM-StEW applies DCCA on sequential latent representations produced by Bi-GRU and Transformer encoders (Rajan et al., 2020, Rajan et al., 2021). Another family matches distributions rather than sample pairs: Soft Alignment for end-to-end speech translation uses a modality discriminator and adversarial training so that speech-derived and text-derived representations become indistinguishable at the space level without collapsing task-specific structure (Zhang et al., 2023). A third family uses optimal transport. CMOT aligns speech and text tokens through a relaxed OT problem with a window constraint, and VTaMo uses entropy-regularized OT with a learnable null token to obtain fine-grained frame-to-token correspondences in sign language translation (Zhou et al., 2023, Hu et al., 10 Jul 2026).

Several works combine local and global alignment. AlignMamba uses relaxed optimal transport for token-level alignment and Maximum Mean Discrepancy for global distribution alignment before multimodal fusion with a Mamba backbone (Li et al., 2024). VTaMo combines local OT, global Earth Mover’s Distance under a learnable orthogonal transformation, and position-aligned contrastive learning (Hu et al., 10 Jul 2026). CL2CM likewise pairs sentence-level InfoNCE with word-level OT-based alignment and relational knowledge distillation from a cross-lingual teacher network (Wang et al., 2023). These designs indicate that alignment is often most effective when both correspondence structure and global geometry are constrained simultaneously.

A more abstract formulation appears in the linear-algebraic treatment of perfect alignment. There, paired modality matrices are stacked into a multimodal data matrix $X$, and perfect alignment is posed as the inverse problem

$AX=\mathbf{0},$

with the rows of $A$ drawn from the left nullspace of $X$ when that nullspace has sufficient dimension (Kamboj et al., 19 Mar 2025). When exact matching is impossible, the smallest-singular-value directions of the SVD provide an approximate solution (Kamboj et al., 19 Mar 2025). This suggests a limiting case of CMTA in which alignment is not merely encouraged by a loss, but solved in closed form under linear assumptions.

3. Recurrent architectural families

The literature repeatedly returns to a small number of architectural templates.

Family	Representative papers	Characteristic mechanism
Asymmetric stronger-to-weaker transfer	(Rajan et al., 2020, Rajan et al., 2021)	Weaker→stronger translation plus latent correlation alignment
Distribution-level modality invariance	(Zhang et al., 2023)	Adversarial speech–text space alignment without pairwise collapse
OT-guided sequence alignment	(Zhou et al., 2023, Hu et al., 10 Jul 2026)	Token or frame transport, often with monotonic or null-token structure
Dual translation with reciprocal recalibration	(Zhou et al., 2023)	Two encoder–decoder branches and cross-modal attention
Shared bottleneck composition	(Wang et al., 2019)	Encoder–decoder mix-and-match for unseen modality pairs
Structure-pivoted alignment	(Wu et al., 2023, Wang et al., 2023)	Scene graphs, syntax trees, or cross-lingual teacher structure as pivots

The asymmetric family treats one modality as a teacher. In SEW and CM-StEW, the stronger modality is defined operationally as the one whose uni-modal model achieves higher task performance, and all cross-modal supervision is organized around improving the weaker modality’s test-time behavior (Rajan et al., 2020, Rajan et al., 2021). By contrast, pathology–genomics CMTA avoids privileging one modality and uses bidirectional cross-attention plus detached alignment targets so that complementary information is not erased (Zhou et al., 2023). RGB–infrared TSFADet occupies an intermediate position: it uses per-object modality selection to choose the more reliable reference modality before applying translation-scale-rotation alignment to the other stream (Yuan et al., 2022).

Another divide concerns what is being aligned. Some methods align latent vectors directly through $L_1$, MSE, or DCCA-style correlation (Rajan et al., 2020, Zhou et al., 2023). Others align predictive distributions: CMOT uses symmetric KL terms between speech-only, text-only, and mixup-based translation outputs, while CV-SLT uses one KL term between posterior and prior latent distributions and a second KL term for self-distillation between decoder outputs (Zhou et al., 2023, Zhao et al., 2023). Still others align graph- or token-structured objects. Cross2StrA aligns scene-graph nodes across image and pivot language and constituency-tree nodes across pivot and target languages (Wu et al., 2023); VTaMo aligns sign-video segments and pseudo-gloss tokens at the transport-plan level (Hu et al., 10 Jul 2026).

These architectural choices reflect different assumptions about modality gaps. Hard pairwise matching presumes that corresponding units should be nearly coincident. Distributional alignment presumes that modality-specific microstructure should survive, provided the overall geometry is made compatible. Structure-pivoted systems presume that alignment is easiest after projecting raw modalities into semantically or syntactically typed intermediate objects.

4. Application domains

Affect recognition is one of the clearest demonstrations of CMTA’s original motivation: multimodal training with uni-modal testing. On RECOLA, SEW uses audio, video-geometric, and video-appearance features for continuous arousal and valence prediction, while CM-StEW extends the same idea with Bi-GRU and Transformer encoders to exploit temporal context on both RECOLA and CMU-MOSI (Rajan et al., 2020, Rajan et al., 2021). In both cases, the central claim is not that multimodal inference is improved, but that a weaker uni-modal system can inherit task-relevant structure from a stronger modality while remaining deployable when the stronger modality is unavailable.

Speech and speech-related translation form a second major cluster. T-Modules frames zero-shot cross-modal transfer between speech and text as decoding from multilingual multimodal fixed-size representations and reports competitive results on several text and speech translation tasks, including significant improvement of the state of the art for zero-shot speech translation on MuST-C (Duquenne et al., 2022). CMOT addresses end-to-end speech translation under limited data by aligning speech and text with optimal transport and mixing the modalities at token level (Zhou et al., 2023). Soft Alignment rethinks the same speech–text gap and argues that space-level adversarial alignment preserves machine translation quality better than hard speech–text alignment (Zhang et al., 2023).

Vision–language and multilingual retrieval systems expose a different use case: exploiting alignment as an alternative to direct supervision. CL2CM learns source–target cross-lingual structure and transfers it to vision–target-language alignment for multilingual image-text and video-text retrieval, operating without human-annotated vision–target-language pairs (Wang et al., 2023). Cross2StrA addresses unpaired cross-lingual image captioning by composing image-to-pivot captioning with pivot-to-target translation and adding scene-graph and constituency-tree alignment plus cross-modal and cross-lingual back-translation (Wu et al., 2023).

Biomedical and scientific applications further broaden CMTA’s scope. In survival analysis, CMTA links pathology whole-slide images and grouped genomic profiles through bidirectional cross-modal attention, reciprocal translation, and alignment-constrained recalibration before discrete-time survival prediction (Zhou et al., 2023). In RGB–infrared aerial vehicle detection, TSFADet models weak cross-modal misalignment through translation, scale, and rotation deviations at the oriented RoI level and shows that region-wise feature alignment is preferable to assuming a single global registration (Yuan et al., 2022).

Sign language translation reveals perhaps the strongest recent shift toward explicit multi-granular alignment. CV-SLT uses a conditional variational autoencoder with a prior path based on visual information only and a posterior path based on visual plus textual information, with one KL term for latent regularization and a second for decoder-level self-distillation (Zhao et al., 2023). VTaMo pushes further by combining local OT with a null token, global orthogonal alignment with EMD, and position-aligned contrastive learning on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL (Hu et al., 10 Jul 2026).

5. Empirical regularities and contested design choices

Across the literature, translation and alignment repeatedly appear as complementary rather than interchangeable. In SEW, removing both the inter-modal translation decoder and CCA alignment makes performance collapse toward the uni-modal baseline, while removing only CCA or only the translation decoder causes noticeable degradation (Rajan et al., 2020). CM-StEW reports the same pattern on sequential affect tasks: decoder-based translation is especially important for arousal on visual modalities, whereas latent alignment is particularly important for valence on acoustic input (Rajan et al., 2021). These results indicate that translation supplies cross-modal content, while alignment determines whether that content becomes usable in the downstream latent geometry.

A second recurring finding is that hard pairwise alignment can be too strong. In Soft Alignment for speech translation, both hard alignment and soft alignment improve speech translation by roughly two BLEU on average, but hard alignment catastrophically harms machine translation, including a drop of up to $-9.6$ BLEU on En–Fr with external MT data, whereas soft adversarial alignment preserves or slightly improves MT and causes only negligible ASR degradation (Zhang et al., 2023). This directly challenges the frequent intuition that closer pairwise matching is always better.

Optimal transport has emerged as a particularly important tool, but its role is nuanced. CL2CM shows that OT-based pseudo-labels are more robust than cross-attention-based pseudo-labels under noisy machine translation in cross-lingual cross-modal retrieval (Wang et al., 2023). CMOT shows that relaxed OT with a window improves both BLEU and alignment score over multitask baselines in speech translation, yet adding OT as an explicit training loss does not help (Zhou et al., 2023). VTaMo’s ablations show that local OT, global EMD, and contrastive learning are complementary, with sizeable BLEU-4 drops when any of them is removed (Hu et al., 10 Jul 2026). A plausible implication is that OT is often most effective as a correspondence engine and geometric prior, rather than as the sole optimization objective.

The same pattern appears in zero-pair and biomedical settings. Mix-and-match networks show that autoencoders, latent consistency, and robust side information are essential for unseen modality pairs, and that pseudo-pairs can recover shared information between unseen modalities that is not reachable through the anchor modality alone (Wang et al., 2019). In survival analysis, removing the alignment constraint reduces C-index, but removing tensor detaching causes large performance collapse because the model converges to redundant shared information and loses complementary prognostic signals (Zhou et al., 2023). These findings counter a second misconception: that maximizing shared information is automatically desirable. Much of CMTA’s practical difficulty lies in deciding what should align and what should remain modality-specific.

6. Open problems and future directions

Several limitations recur with striking consistency. Many CMTA methods require paired or temporally aligned data at training time. SEW and CM-StEW need co-occurring stronger and weaker modalities (Rajan et al., 2020, Rajan et al., 2021); mix-and-match networks assume at least anchor-paired datasets (Wang et al., 2019); pathology–genomics CMTA depends on paired WSIs and genomic profiles (Zhou et al., 2023). This suggests that the hardest open problem is not merely better alignment under supervision, but reliable CMTA under sparse, noisy, or partially missing correspondence.

Scalability is another pressure point. OT-based systems must control transport cost computation over long sequences or large vocabularies; CL2CM explicitly notes that OT over long sentences or large vocabularies can be costly (Wang et al., 2023), and VTaMo introduces windowing, entropy annealing, and memory queues to stabilize and contain that cost (Hu et al., 10 Jul 2026). Large multimodal systems also face retraining costs. CACARA proposes an emergent alignment strategy in which a new modality is aligned only to a frozen multilingual text encoder, yielding multilingual audio–text support without full retraining and with substantially lower training time, energy, and emissions than fully tri-modal training (Moreira et al., 29 Nov 2025). This suggests a promising direction in which CMTA becomes incremental and text-centric rather than jointly retrained for every new modality.

Theoretical clarity remains limited outside simplified settings. The perfect-alignment formulation provides exact nullspace conditions under linear assumptions, but its own discussion emphasizes linearity, paired-data dependence, non-uniqueness of the recovered latent, and the lack of guaranteed generalization to unseen data (Kamboj et al., 19 Mar 2025). At the other extreme, deep CMTA systems often work well empirically but offer weak guarantees about when alignment will preserve class structure, causal semantics, or calibration under domain shift. Bridging these two ends—closed-form geometry and high-capacity representation learning—remains unresolved.

Finally, there is an unresolved tension between explicit intermediate structure and end-to-end simplicity. CV-SLT explicitly points to discrete latent variables as a possible future route to better intermediate representations in gloss-free sign language translation (Zhao et al., 2023). Cross2StrA shows that scene graphs and constituency trees can materially improve relevancy and fluency in unpaired cross-lingual captioning (Wu et al., 2023). Yet soft-alignment and emergent-alignment results suggest that not every task benefits from ever more rigid structure (Zhang et al., 2023, Moreira et al., 29 Nov 2025). This suggests that the next phase of CMTA research may be defined less by a single favored loss and more by selective explicitness: making correspondence structure explicit where it stabilizes learning, while keeping the shared space flexible enough to preserve modality-specific information and adapt to missing or newly added modalities.