Bidirectional Cross-modal Grouped Alignment

• Bidirectional cross-modal grouped alignment is a multimodal strategy that aligns structured groups across heterogeneous modalities while preserving modality-specific features.
• It employs mechanisms such as adaptive thresholding and dual attention to construct token or patch groups from fine-grained similarity data.
• Empirical results demonstrate that this approach improves performance, robustness, and diagnostic accuracy in medical image-report and other cross-modal tasks.

Bidirectional cross-modal grouped alignment denotes a class of multimodal learning strategies that align representations in both directions across modalities while using grouped units rather than only global instance embeddings. In current formulations, those groups may be gene sets and pathology-associated summaries, token-grouped visual embeddings and patch-grouped language embeddings, pathological regions, temporal segments, perturbation-defined sample groups, discrete code indices, or shared sparse-neuron groups. Across these settings, the central objective is to model cross-modal correspondence without erasing modality-specific structure, typically by combining fine-grained grouping, symmetric or quasi-symmetric alignment objectives, and mechanisms that prevent trivial collapse into a single undifferentiated latent space (Li et al., 31 Jul 2025, Zhou et al., 2023, Wang et al., 2022).

1. Conceptual basis

The term has three technically distinct components. “Cross-modal” refers to alignment across heterogeneous representations such as pathology and genomics, medical images and reports, video and language, fMRI and video, audio and text, or weakly paired omics modalities. “Bidirectional” means that each modality acts as both source and target, as in language-to-vision and vision-to-language group alignment in AGA, pathology-to-genomics and genomics-to-pathology translation in CMTA, or symmetric anchor roles in GroupCLIP (Li et al., 31 Jul 2025, Zhou et al., 2023, Gorla et al., 3 Feb 2026). “Grouped” indicates that the aligned units are structured subsets rather than only whole-sample embeddings: token-centered visual groups and patch-centered language groups, disease prototypes, region masks, multi-scale cortical levels, discrete codebook indices, perturbation-label groups, or dictionary-neuron groups (Li et al., 31 Jul 2025, Wang et al., 2022, You et al., 4 Jan 2026, Sen et al., 12 May 2026, Kaushik et al., 27 Jan 2026).

Recent work has broadened the notion of a group well beyond explicit segmentation. In AGA, each token selects its top-matching patches to form a token-grouped visual representation, and each patch selects its most related tokens to form a patch-grouped language representation (Li et al., 31 Jul 2025). In CMTA, the genomic side is already grouped into biologically defined gene groups, while cross-modal attention induces genomics-indexed pathology summaries and pathology-indexed genomic summaries (Zhou et al., 2023). In CoDAAR, the group is a code index shared across modality-specific codebooks, so index $k$ in audio, video, and text is intended to correspond to the same semantic group while retaining modality-specific structure (Sen et al., 12 May 2026). In group-sparse multimodal autoencoders, a group is the pair of latent coefficients attached to the same neuron index across paired modalities, and alignment is expressed as joint sparsity over those paired coefficients (Kaushik et al., 27 Jan 2026).

A recurring consequence of this shift from global to grouped units is that alignment becomes hierarchical. NeuroAlign explicitly separates global temporal-semantic alignment from fine-grained pattern matching over multi-scale temporal groups and shared discrete codes (You et al., 4 Jan 2026). MGCA for medical image-report learning decomposes alignment into pathological region-level, instance-level, and disease-level correspondences (Wang et al., 2022). The open-vocabulary segmentation MGCA likewise operates simultaneously at object, region, and pixel granularity (Liu et al., 2024). This suggests that “grouped” is not merely a batching choice; it is a representational commitment that the relevant semantics are internally structured.

2. Group construction and correspondence mechanisms

A canonical grouped construction appears in AGA, where fine-grained token–patch similarity is first computed as

$$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$

then thresholded into a sparse matrix

$$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$

This sparse matrix defines bidirectional groups. For each token $t_m$, the token-grouped visual representation is

$p_m = \sum_{r=1}^R \alpha_{mr} v_r,$

and for each patch $v_n$, the patch-grouped language representation is

$q_n = \sum_{m \in \mathcal{G}(n)} \beta_{nm} t_m.$

AGA then makes the grouping adaptive through the Language Grouped Threshold Gate and Vision Grouped Threshold Gate, replacing a fixed threshold with running, learnable thresholds $\sigma_{tg}$ and $\sigma_{vg}$ (Li et al., 31 Jul 2025). The key point is that group membership is data-dependent, direction-dependent, and instance-local.

CMTA uses a different but closely related mechanism. Starting from pathology instance tokens $\slashed{P}$ and genomic instance tokens $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$0, it defines bidirectional cross-modal attention maps

$$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$1

which induce grouped summaries

$$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$2

Here the groups are not produced by clustering or thresholding but by attention rows: each row of $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$3 is a soft pathology group attached to a genomic group, and each row of $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$4 is a soft genomic group attached to a pathology patch (Zhou et al., 2023). The paper explicitly characterizes this dual attention as a bidirectional information bridge.

Other domains instantiate group construction through domain-specific surrogates. In CMAC for audio-video representation learning, groups are implicit positions in visual spatiotemporal feature maps and audio time–frequency feature maps; audio-guided visual attention and visual-guided audio attention define group-level importance over these local units (Min et al., 2021). In open-vocabulary segmentation MGCA, pseudo object and region groups are formed by top-$$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$5 selection over pixel–text similarity, and adaptive semantic units at inference group pixels around meta-points into part-level semantic units (Liu et al., 2024). In CroBIM, cross-scale regions with high attention deficit are grouped and jointly refined, so grouping is driven by disagreement across scales rather than by fixed proposals (Dong et al., 2024).

Two general design choices recur. First, grouped units are usually induced inside each paired sample rather than across the dataset, which avoids reliance on globally stable region vocabularies. Second, grouping is often asymmetric at construction time: tokens select patches, patches select tokens; gene groups attend to pathology patches, pathology patches attend to gene groups. Bidirectionality therefore begins before the loss function, at the level of how candidate correspondences are generated.

3. Bidirectionality and optimization regimes

The most straightforward bidirectional regime is symmetric contrastive learning. AGA uses two group-level losses, $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$6 and $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$7, after cross-attention between token-grouped visual embeddings and patch-grouped language embeddings, and each loss itself contains forward and reverse InfoNCE-style terms within the same image-text pair (Li et al., 31 Jul 2025). CRA uses a bidirectional contrastive alignment loss between grounded video features $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$8 and pooled QA features $$S = [s_{mn}]_{M \times N}, \qquad s_{mn} = v_n \cdot t_m,$$9,

$$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$0

with $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$1 and $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$2, so grounding and question reasoning are jointly constrained in both directions (Chen et al., 5 Mar 2025). MGCA for medical image-report learning similarly uses symmetric image-to-text and text-to-image InfoNCE at the instance level and symmetric token-wise alignment at the local level (Wang et al., 2022).

A second regime is bidirectional translation with asymmetric gradient flow. CMTA is exemplary. It builds two translations, $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$3 and $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$4, then enforces

$$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$5

However, the intra-modal targets $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$6 and $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$7 are detached from the computational graph when $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$8 is computed. The paper states that otherwise the model converges to redundant shared information and fails to predict survival events (Zhou et al., 2023). This is a central technical caution: a method may be bidirectional in translation space while still requiring one-way optimization within each directional term.

A third regime aligns distributions, prototypes, or discrete groups rather than point embeddings. MGCA’s cross-modal prototype alignment uses Sinkhorn-derived cluster assignments and enforces cross-modal cluster assignment consistency between images and reports, thereby aligning disease-level groups rather than only individual samples (Wang et al., 2022). CoDAAR uses modality-specific codebooks with a shared index set and aligns them through Discrete Temporal Alignment, Cascading Semantic Alignment, cross-modal commitment, and CMCM loss, so code index $$\widetilde{s}_{jk} = \begin{cases} \widehat{s}_{jk}, & \text{if } \widehat{s}_{jk} \ge \sigma,\ 0, & \text{otherwise}. \end{cases}$$9 acts as the shared group identity across modalities (Sen et al., 12 May 2026). NeuroAlign likewise uses a shared discrete codebook plus structural alignment of pairwise similarity matrices across fMRI and video codes (You et al., 4 Jan 2026).

Weakly paired settings motivate yet another optimization pattern. GroupCLIP in GROOVE defines positives for an anchor $t_m$0 as all samples in the other modality with the same perturbation label: $t_m$1 Averaging this over both modalities yields a fully bidirectional group-level contrastive objective even without instance-level pairs (Gorla et al., 3 Feb 2026). In group-sparse multimodal SAEs, the grouped penalty is instead

$t_m$2

which encourages paired samples from different modalities to activate the same latent neuron groups (Kaushik et al., 27 Jan 2026).

A final variant is indirect semantic alignment under constrained decoupling. CDDS separates semantic and modality components, then aligns sampled cross-modal semantic surrogates $t_m$3 with $t_m$4 and $t_m$5 with $t_m$6, rather than directly forcing $t_m$7 and $t_m$8 together. Its semantic consistency term is explicitly symmetric across modalities, while its distribution-sampling step is meant to avoid semantic deviation caused by modality gaps (Ma et al., 5 Mar 2026). This suggests that bidirectionality need not be implemented only as symmetric InfoNCE; it can also be realized through symmetric reconstruction, symmetric prototype prediction, symmetric backtranslation, or symmetric decoupled alignment.

4. Representative instantiations across domains

Work	Modalities	Grouped bidirectional mechanism
CMTA (Zhou et al., 2023)	Pathology WSIs + grouped genomics	Dual encoder–decoder translation, bidirectional attention, two-way alignment loss
AGA (Li et al., 31 Jul 2025)	Medical images + reports	TGV/PGL groups, adaptive threshold gates, BCGA
MGCA (Wang et al., 2022)	Medical images + reports	Region-, instance-, and disease-level alignment
CRA (Chen et al., 5 Mar 2025)	Video + QA text	Grounded segment ↔ QA bidirectional contrastive alignment
CroBIM (Dong et al., 2024)	Remote sensing image + expression	Cascaded bidirectional cross-attention over multi-scale grouped tokens
NeuroAlign (You et al., 4 Jan 2026) / CoDAAR (Sen et al., 12 May 2026)	fMRI-video-text / audio-video-text	Hierarchical groups, shared discrete codes, bidirectional temporal or semantic alignment
GROOVE (Gorla et al., 3 Feb 2026) / multimodal SAE (Kaushik et al., 27 Jan 2026)	Weakly paired omics / aligned embeddings	Group-label positives or shared neuron groups across modalities

In survival analysis, CMTA is a prototypical translation-and-recalibration framework. It criticizes naïve concatenation and uni-directional guidance, then uses two parallel encoder–decoder paths to produce intra-modal and cross-modal patient-level embeddings, with cross-modal attention linking grouped genomic and pathology units (Zhou et al., 2023). In medical vision–language pretraining, AGA is the most literal realization of “Bidirectional Cross-modal Grouped Alignment”: token-grouped visual representations, patch-grouped language representations, within-pair alignment, and BCGA based on cross-attention between these group embeddings (Li et al., 31 Jul 2025).

MGCA for generalized medical visual representation learning broadens the idea into a three-level scheme. It combines instance-wise bidirectional contrastive alignment, token-wise bidirectional cross-attention, and disease-level cluster assignment consistency through prototypes (Wang et al., 2022). The open-vocabulary segmentation MGCA makes the same multi-grained principle operational for dense prediction, explicitly learning object-, region-, and pixel-level image–text alignment despite training only on image–caption pairs (Liu et al., 2024).

Outside medical data, CRA uses bidirectional alignment between grounded video segments and QA features, embedding grouped alignment into weakly supervised video question grounding (Chen et al., 5 Mar 2025). CroBIM places bidirectional interaction inside a segmentation architecture via vision-guided language refinement followed by language-guided pixel refinement, with grouped multi-scale visual tokens and region-focused compensation modules (Dong et al., 2024). NeuroAlign and CoDAAR move the field toward hierarchical and discrete formulations: temporal segments, cortical levels, and codebook entries become the aligned groups, rather than explicit regions or tokens (You et al., 4 Jan 2026, Sen et al., 12 May 2026). GROOVE and multimodal group-sparse autoencoders show that the same principle persists when pairing is weak or when alignment is performed over already aligned embedding spaces rather than raw inputs (Gorla et al., 3 Feb 2026, Kaushik et al., 27 Jan 2026).

5. Empirical patterns and recurrent findings

Several empirical regularities recur across otherwise dissimilar architectures. First, explicit grouped cross-modal interaction is usually better than naïve global fusion. CMTA reports c-index improvements over DualTrans across all five TCGA datasets, including +3.03% on BLCA, +1.58% on LUAD, and +2.51% on UCEC, and it also outperforms MCAT, especially on UCEC (+6.39%). Its ablations further show that removing cross-modal attention drops c-index by 1.26–4.93%, removing alignment constraints drops performance by 0.58–3.75%, and removing tensor detaching causes performance degradation up to −9.08% on BLCA (Zhou et al., 2023). The pattern is not merely that “more interaction helps”; rather, interaction must be constrained to avoid collapse.

Second, adaptive grouped alignment materially improves medical image–report learning. In AGA, removing BCGA while keeping IGA sharply reduces retrieval performance: on CheXpert 5×200, Precision@5 drops from 50.28 to 34.28, and on SMTs 3×200 it drops from 55.00 to 45.33. Replacing adaptive thresholding with fixed thresholds also reduces performance, from 50.28 to 48.54 on CheXpert and from 55.00 to 51.00 on SMTs 3×200 (Li et al., 31 Jul 2025). The ablations directly support the claim that the group-to-group bidirectional module, not only within-pair grouping, is the dominant source of gain.

Third, multi-granularity matters when dense outputs are required. In medical MGCA, adding CTA to ITA raises SIIM Dice from 25.0 to 47.6 under 1% labels, while CPA chiefly improves classification AUROC, showing complementary effects of local and grouped disease-level alignment (Wang et al., 2022). In open-vocabulary segmentation MGCA, the full object + region + pixel formulation yields larger gains than any single granularity alone, and replacing pixel units with adaptive semantic units increases average mIoU from 37.1 to 38.8 on datasets with background and from 33.7 to 35.5 on datasets without background (Liu et al., 2024).

Fourth, grouped bidirectional alignment often improves robustness or faithfulness rather than only raw similarity. CRA reports that removing CMA decreases Acc@GQA by 2.0% and [email protected] by 2.2% on NextGQA, indicating that bidirectional segment–QA alignment improves grounded reasoning rather than only answer classification (Chen et al., 5 Mar 2025). GROOVE’s ablations show that removing GroupCLIP causes the largest drop in matching and imputation quality, larger than removing backtranslation alone, so group-level alignment is the key driver even in weakly paired multimodal biology (Gorla et al., 3 Feb 2026). In multimodal group-sparse autoencoders, GSAE and MGSAE substantially outperform ordinary SAEs on zero-shot cross-modal tasks, indicating that shared neuron groups preserve semantic alignment better than split dictionaries (Kaushik et al., 27 Jan 2026).

Finally, a smaller modality gap is not sufficient. The retrieval study on multimodal feature alignment shows that BLIP has the smallest centroid- and Wasserstein-based modality gaps, but also states that smaller modality gap does not guarantee better retrieval performance. It further finds that cosine similarity consistently outperforms Euclidean, Manhattan, Chi-square, and learned MLP-based metrics, while conventional MLP architectures are insufficient for capturing the complex interactions between image and text representations (Xu et al., 10 Jun 2025). This point corrects a common overinterpretation of alignment diagnostics: closeness of distributions and retrieval discriminability are not identical objectives.

6. Limitations, misconceptions, and open directions

A common misconception is that bidirectionality simply means using two symmetric losses. CMTA shows that two-way translation can still require directional gradient blocking to preserve modality-specific prognostic information (Zhou et al., 2023). A second misconception is that better global alignment metrics automatically imply better cross-modal functionality. The retrieval study explicitly observes that Wasserstein distance can be informative as a measure of modality gap, yet conventional architectures such as multilayer perceptrons remain insufficient for feature alignment tasks, and cosine similarity remains the strongest practical metric (Xu et al., 10 Jun 2025). A third misconception is that grouped alignment necessarily requires explicit annotations. Several papers construct groups from sparse similarities, attention maps, or code assignments without dense labels (Li et al., 31 Jul 2025, Liu et al., 2024, Sen et al., 12 May 2026).

The main technical limitations are also recurrent. AGA notes that its evaluation focused on visual representations and did not test text-side downstream tasks, and it assumes paired image–report data while introducing non-trivial per-instance $t_m$9 similarity computation and per-pair cross-attention (Li et al., 31 Jul 2025). GROOVE finds that there is not yet an aligner that uniformly dominates across settings or modality pairs, which means that representation learning and downstream alignment remain entangled design problems (Gorla et al., 3 Feb 2026). CoDAAR requires paired, temporally aligned multimodal sequences and a fixed codebook size $p_m = \sum_{r=1}^R \alpha_{mr} v_r,$0, so the semantic granularity is discretized in advance (Sen et al., 12 May 2026). CDDS identifies the absence of an established standard for distinguishing semantic and modal information and pays a computational price for distribution-level correlation estimation (Ma et al., 5 Mar 2026).

Open directions in the literature are correspondingly structured. AGA proposes extending grouping to the sample level and integrating generation-based pretraining (Li et al., 31 Jul 2025). CoDAAR and NeuroAlign indicate that grouped alignment can be generalized to more than two modalities and to hierarchical codebooks or temporal scales (Sen et al., 12 May 2026, You et al., 4 Jan 2026). GROOVE suggests that weak supervision via labels can define groups even when direct pairs do not exist (Gorla et al., 3 Feb 2026). CDDS suggests that future grouped alignment may profit from explicit semantic–modality decoupling before cross-modal interaction (Ma et al., 5 Mar 2026). Taken together, these works point toward a broader formulation in which group choice, directionality of optimization, and preservation of modality-specific structure are co-equal design variables rather than secondary implementation details.