Cross-Modal Conformer Architectures

• Cross-Modal Conformer is a family of architectures that integrate cross-modal interactions within or adjacent to Conformer blocks to enable effective multimodal fusion.
• Key approaches include integrated in-block cross-modal attention, fusion-around-Conformer with separate unimodal encoders, and external contextualization with alignment objectives.
• These methods improve performance in applications such as audio-visual speech recognition, wake word spotting, and continuous sign language recognition by leveraging synchronized and asynchronous modality interactions.

Cross-Modal Conformer denotes a family of architectures that retain the Conformer as a primary sequence model while introducing cross-modal interaction, multimodal fusion, or modality-conditioned contextualization. The term has no single canonical meaning across the literature. In some works it refers narrowly to Conformer blocks that contain explicit cross-modal attention, such as frame-level audio-visual interaction in wake word spotting or query-swapped attention across RGB and heatmap streams in continuous sign language recognition (Wang et al., 2024, Aloysius et al., 2024). In other works it refers more broadly to Conformer-based systems in which modality-specific encoders are fused by concatenation and a shared Conformer stack, or to Conformer backbones augmented by external speech-text contextual modules and alignment objectives rather than internally cross-modalized blocks (Burchi et al., 2023, Wei et al., 2023, Lu et al., 2023). A precise reading therefore depends on where cross-modal coupling occurs: inside Conformer blocks, between unimodal encoders and a shared Conformer, or alongside the Conformer through auxiliary alignment and conditioning pathways.

1. Conceptual scope and taxonomy

The literature distinguishes at least three technically different senses of Cross-Modal Conformer.

First, there is the integrated cross-modal-block sense, where cross-modal interaction is inserted into each encoder block. The clearest audio-visual example is the Frame-Level Cross-Modal Attention module, which is added to each Conformer block and performs attention over the modality axis at each synchronized frame in audio-visual wake word spotting (Wang et al., 2024). A related vision-language-signing variant is Cross-Modal Relative Attention, where RGB and heatmap streams exchange query vectors inside the attention mechanism of Conformer-based sequence learners (Aloysius et al., 2024).

Second, there is the fusion-around-Conformer sense, where unimodal frontends or unimodal Conformer-style encoders operate separately and the modalities interact only after temporal alignment and feature projection. The Audio-Visual Efficient Conformer for robust ASR is representative: it has an audio encoder, a visual encoder, an audio-visual fusion module, and an audio-visual encoder, with fusion realized by temporally aligned feature concatenation followed by a feed-forward projection rather than explicit cross-attention (Burchi et al., 2023). Earlier end-to-end audio-visual speech recognition with Conformers also uses separate audio and visual Conformer encoders followed by concatenation and an MLP, which is multimodal but not cross-attention-based (Ma et al., 2021).

Third, there is the externally conditioned Conformer sense, where the Conformer itself remains unimodal, but cross-modal information is injected through auxiliary branches, alignment losses, or decoder conditioning. Conversational ASR by learning audio-textual cross-modal contextual representation keeps the Conformer as the acoustic encoder for the current utterance while adding a separate cross-modal extractor and CVAE-based conversational representation module, with the decoder conditioned on local cross-modal context and latent role/topic variables (Wei et al., 2023). Cross-modal alignment with optimal transport for CTC-based ASR similarly preserves a Conformer acoustic encoder but augments it during training with a BERT branch and OT-based speech-text alignment, then discards the text branch at inference while retaining greedy CTC decoding (Lu et al., 2023).

This suggests that “Cross-Modal Conformer” is best treated as a research category rather than a single architecture. A strict usage should be reserved for models whose Conformer blocks themselves implement inter-modal interaction. A broader usage includes Conformer-centered multimodal systems in which fusion, alignment, or conditioning occurs adjacent to rather than inside the Conformer.

Pattern	Representative mechanism	Representative papers
Integrated cross-modal block	Cross-modal attention inside each Conformer block	(Wang et al., 2024, Aloysius et al., 2024)
Fusion around Conformer	Separate encoders, aligned concatenation, shared Conformer or MLP fusion	(Burchi et al., 2023, Ma et al., 2021)
External contextualization	Auxiliary speech-text extractor, OT alignment, conditional decoder	(Wei et al., 2023, Lu et al., 2023)

2. Core architectural motifs

Across application domains, Cross-Modal Conformer designs typically begin with modality-specific frontends that normalize sampling rate, spatial structure, or feature geometry before any cross-modal interaction occurs. In audio-visual ASR, one stream may start from raw waveform or log-mel features and the other from cropped lip regions processed by a 3D convolution stem and ResNet-18, after which Conformer or Efficient Conformer backends operate on modality-specific temporal sequences (Burchi et al., 2023, Ma et al., 2021). In CSLR, RGB and heatmap videos are first encoded by S3D backbones and projected to 512-dimensional frame embeddings before entering Conformer sequence learners (Aloysius et al., 2024).

A recurring design choice is temporal synchronization before interaction. In AVWWS, the audio frontend downsamples 256 audio frames to 64 so that the audio sequence matches the 64-frame lip sequence, enabling frame-level cross-modal attention over the modality dimension (Wang et al., 2024). In AVEC, the audio backend downsamples a 20 ms stream to 80 ms, and the visual backend downsamples a 40 ms stream to the same 80 ms rate, enabling direct concatenation-based fusion prior to a shared audio-visual encoder (Burchi et al., 2023). By contrast, the cross-attention conformer for context modeling in speech enhancement is explicitly designed for unequal-length sequences and does not require synchronization: the noisy speech sequence provides queries and the auxiliary noise-context sequence provides keys and values (Narayanan et al., 2021).

The Conformer backbone itself is generally retained in recognizably standard form. Where equations are provided, the block consists of the usual Macaron-style dual FFN structure with MHSA and a convolution module in between, followed by normalization (Wei et al., 2023, Lu et al., 2023). Other papers rely on the standard Conformer conceptually without reprinting full equations, but still describe the same module stack of feed-forward, self-attention, convolution, and feed-forward components (Ma et al., 2021, Wang et al., 2024). The cross-modal novelty therefore usually lies not in replacing the Conformer wholesale, but in modifying what the attention sees, what gets fused, or how contextual representations are injected.

A further motif is hierarchical or staged efficiency. AVEC inherits Efficient Conformer, organizing encoders into stages with progressive temporal downsampling and widening feature dimensions, and introduces patch attention in the first audio backend stage as a cheaper substitute for grouped attention (Burchi et al., 2023). This indicates that Cross-Modal Conformer work often combines multimodal design with the efficiency concerns already present in Conformer research.

3. Modes of cross-modal interaction

The most explicit cross-modal mechanism in the surveyed literature is attention over the modality axis. In FLCMA-based AV-Conformer, the multimodal tensor is written as $$X_{av} = [X_{spec}, X_{lip}] \in \mathbf{R}^{M \times T \times D}$$ with $M=2$, and attention is computed per time frame across the two modalities rather than across time. Because both audio and visual representations contribute to $Q$, $K$, and $V$, the mechanism is symmetric cross-modal mixing within each synchronized frame, while temporal MHSA and the Conformer convolution module remain responsible for global and local temporal modeling, respectively (Wang et al., 2024).

A different but related mechanism is query exchange while preserving modality-specific memory. In CMRA for CSLR, the RGB stream computes

$$attention_{rgb} = attention(Q_{rgb},K_{rgb},V_{rgb}) + attention(Q_{hm},K_{rgb},V_{rgb}),$$

and the heatmap stream computes

$$attention_{hm} = attention(Q_{hm},K_{hm},V_{hm}) + attention(Q_{rgb},K_{hm},V_{hm}).$$

This is not simple early fusion and not fully shared self-attention. Instead, each modality keeps its own keys and values, while the other modality’s queries influence what is retrieved from that modality’s memory (Aloysius et al., 2024).

A broader class uses feature-level concatenation followed by projection rather than attention. AVEC aligns unimodal encoded states $\mathbf{a}_t$ and $\mathbf{v}_t$, concatenates them as $[\mathbf{a}_t;\mathbf{v}_t]$, and applies a feed-forward projection

$M=2$0

This is more than late fusion of logits, because interaction begins at aligned intermediate features, but it is less than explicit cross-attention because the modalities do not attend to each other before concatenation (Burchi et al., 2023). The earlier end-to-end AVSR Conformer uses a closely related pattern: modality-specific Conformer encoders, concatenation, and an MLP back to model dimension (Ma et al., 2021).

Another mode is asymmetric query-to-context attention. In the cross-attention conformer for speech enhancement, the main speech sequence $M=2$1 and noise context $M=2$2 are processed separately by FFN and convolution, then the model forms

$M=2$3

applies FiLM fusion,

$M=2$4

and performs a second cross-attention step into the fused representation before the final FFN and normalization (Narayanan et al., 2021). This architecture is not multimodal in the experimental sense, since both streams are audio-derived, but it establishes a generic Conformer-compatible recipe for conditioning one sequence on another of different length.

Finally, some systems perform cross-modal interaction outside the Conformer block. Conversational ASR with audio-textual cross-modal contextual representation uses a separate 3-layer Transformer cross-modal encoder that operates on concatenated speech and text features, while the Conformer encodes only the current utterance. The decoder then attends to or linearly fuses the resulting context vectors (Wei et al., 2023). Cross-modal alignment with OT likewise adds a text branch and transport-based supervision above the final Conformer encoder layer rather than altering internal Conformer attention (Lu et al., 2023).

4. Optimization, alignment, and training objectives

Cross-Modal Conformer systems span several objective families, and the chosen loss often reveals what kind of cross-modal coupling the architecture is designed to learn.

For CTC-centered speech systems, intermediate supervision is prominent. AVEC inserts intermediate CTC losses between selected Conformer blocks in the audio backend, visual backend, and audio-visual encoder. The intermediate prediction is

$M=2$5

and residual conditioning is applied through

$M=2$6

The total loss combines final and intermediate CTC losses,

$M=2$7

with $M=2$8, and the paper reports that this improves optimization and strengthens visual modality usage, particularly when audio is masked (Burchi et al., 2023). In CSLR, the RGB, heatmap, fusion, and APN branches all contribute CTC losses to the total recognition objective

$M=2$9

which makes the multimodal system an ensemble of CTC-supervised branches rather than a single-stream decoder (Aloysius et al., 2024).

For binary detection tasks, weighted BCE is used. The FLCMA-based AVWWS model optimizes weighted Binary Cross-Entropy with negative:positive $Q$0, followed at inference by thresholding of the predicted wake-word probability, while evaluation emphasizes FRR, FAR, WWS score, and AUC (Wang et al., 2024).

For external speech-text contextualization, masked multimodal pretraining and conditional generative objectives appear. The conversational ASR model trains its cross-modal extractor with token masking on both speech and text, a modal-level mask input that replaces an entire modality with zero vectors with probability 30%, and an auxiliary CTC objective. The extractor loss is

$Q$1

after which the extractor is frozen during conversational ASR training. Decoder conditioning then combines local cross-modal context with CVAE-derived latent variables for role preference and topic coherence (Wei et al., 2023).

The most explicit alignment formulation is optimal transport between speech and text representations. In OT-based CTC ASR, a Conformer encoder produces acoustic states $Q$2, which are projected into BERT space as

$Q$3

Given BERT outputs $Q$4, the model solves an entropy-regularized OT problem with cosine cost

$Q$5

obtains a coupling matrix $Q$6, and transports the acoustic sequence into token-length text space via

$Q$7

The final objective combines CTC, OT, and cosine alignment losses,

$Q$8

with $Q$9, $K$0, and $K$1 for entropy regularization (Lu et al., 2023). This is a distinct notion of cross-modality: not joint inference over two modalities, but training-time transfer of text-side contextual knowledge into a Conformer acoustic encoder.

Pretraining also appears in non-speech settings. ConSignformer initializes its Conformer with a denoising-style regression task on Indian Sign Language pose data, predicting clean 443-dimensional gesture descriptors from noisy ones using MSE loss (Aloysius et al., 2024). This suggests that Cross-Modal Conformer research often couples architectural cross-modality with representation pretraining tailored to the task’s latent structure.

5. Representative applications and empirical behavior

The most developed application area is audio-visual speech processing. In robust AVSR, AVEC reports state-of-the-art WER of $K$2 and $K$3 on LRS2 and LRS3 test sets with a neural LM, and emphasizes that audio-visual training reaches lower WER with 4 times fewer training steps than the audio-only model (Burchi et al., 2023). Earlier end-to-end AVSR with modality-specific Conformer encoders and late MLP fusion also reports strong results on LRS2 and LRS3, but the clean-condition gains of audio-visual over audio-only are modest when the audio model is already strong; the paper explicitly argues that visual information becomes most valuable under severe noise (Ma et al., 2021). This pattern recurs across the AV literature: multimodality is most informative when acoustic reliability drops.

In wake word spotting, the case for explicit cross-modal attention is especially direct. On the far-field MISP dataset, the FLCMA-based AV-Conformer with pretraining achieves a $K$4 WWS score, with FRR $K$5, FAR $K$6, and AUC reported as $K$7. The ablation from AV-Conformer(E) at $K$8 WWS to AV-Conformer(E) + FLCMA at $K$9, and then to $V$0 with pretraining, is the clearest evidence in the surveyed papers that frame-level cross-modal attention inside Conformer blocks improves over simple add/concat fusion (Wang et al., 2024).

In conversational ASR, cross-modality is used less for noise robustness than for contextual semantics. The audio-text contextual representation model reports relative accuracy improvements of $V$1 on HKUST and $V$2 on MagicData-RAMC compared to the standard Conformer baseline, with the best system reaching CER $V$3 on HKUST/dev and $V$4 on MagicData-RAMC/test (Wei et al., 2023). The results indicate that short local cross-modal context and longer CVAE-compressed conversational context are complementary, while directly injecting long raw contextual history can hurt.

In CTC ASR with text-side knowledge transfer, OT alignment produces large gains without requiring an external LM at inference. On AISHELL-1, the baseline Conformer+CTC achieves dev/test CER $V$5, whereas ConformerAdpt+CTC-OT-BERT reaches $V$6, corresponding to relative improvements of $V$7 and $V$8 (Lu et al., 2023). This is a different empirical profile from audio-visual fusion papers: the gain comes from training-time linguistic enrichment of the acoustic encoder rather than from an additional input modality at test time.

In speech enhancement for ASR, cross-attention conditioning on a preceding noise-only segment improves downstream recognition under adverse conditions. The best model E3 yields relative WER improvements of about $V$9–$$attention_{rgb} = attention(Q_{rgb},K_{rgb},V_{rgb}) + attention(Q_{hm},K_{rgb},V_{rgb}),$$0 over the no-enhancement baseline in low-SNR settings and about $$attention_{rgb} = attention(Q_{rgb},K_{rgb},V_{rgb}) + attention(Q_{hm},K_{rgb},V_{rgb}),$$1–$$attention_{rgb} = attention(Q_{rgb},K_{rgb},V_{rgb}) + attention(Q_{hm},K_{rgb},V_{rgb}),$$2 over the context-free enhancement model, with gains also observed on multi-talker mixtures despite the model not being trained specifically on such mixtures (Narayanan et al., 2021). Although not strictly multimodal, these results are relevant because they validate cross-attention Conformer blocks for unequal-length conditioning sequences.

In continuous sign language recognition, ConSignformer extends the Cross-Modal Conformer idea to bimodal vision streams. The final model reports WER $$attention_{rgb} = attention(Q_{rgb},K_{rgb},V_{rgb}) + attention(Q_{hm},K_{rgb},V_{rgb}),$$3 on PHOENIX14T dev/test and $$attention_{rgb} = attention(Q_{rgb},K_{rgb},V_{rgb}) + attention(Q_{hm},K_{rgb},V_{rgb}),$$4 on PHOENIX14 dev/test, improving over the paper’s baseline and outperforming the previous best reported test WER in its comparison table (Aloysius et al., 2024). The ablation from pretrained Conformer to pretrained Conformer + CMRA, and then to the full model with APN, supports the claim that cross-modal attention inside the sequence model contributes beyond simple fusion.

6. Technical distinctions, misconceptions, and research directions

A common misconception is that any multimodal Conformer is automatically a cross-modal-attention architecture. The surveyed literature contradicts this. End-to-end AVSR with separate audio and visual Conformer encoders plus concatenation and MLP fusion is multimodal but not cross-attentional (Ma et al., 2021). AVEC is similarly a multimodal Conformer with early or intermediate feature fusion, not a model with audio-to-video or video-to-audio attention heads (Burchi et al., 2023). Strictly speaking, these systems should be classified as multimodal Conformer baselines or fused Conformer architectures rather than fully cross-modalized Conformers.

Another misconception is that cross-modality must imply symmetric bidirectional interaction. Several systems are deliberately asymmetric. The speech enhancement cross-attention conformer treats one sequence as the carrier and the other as contextual memory (Narayanan et al., 2021). OT-based speech-text alignment treats text as a training-time supervisory modality that is absent during inference (Lu et al., 2023). Conversational ASR with a cross-modal extractor likewise uses speech-text supervision during training but relies on speech-only historical inputs at inference by replacing missing text with zero vectors (Wei et al., 2023). This suggests that Cross-Modal Conformer design is often driven less by abstract symmetry than by deployment constraints.

A third distinction concerns alignment assumptions. FLCMA and AVEC depend on synchronization or deliberate resampling to a shared temporal rate (Wang et al., 2024, Burchi et al., 2023). Cross-attention conformer for context modeling avoids such assumptions by using query-key/value attention across unequal-length sequences (Narayanan et al., 2021). OT-based alignment goes further by replacing hard alignment with a transport coupling between distributions of different lengths (Lu et al., 2023). A plausible implication is that the choice between concatenation, modality-axis attention, and cross-sequence attention is fundamentally a choice about the expected temporal relation between modalities.

Several open directions are already implicit in the surveyed papers. Audio-visual work repeatedly identifies the limitations of static concat-based fusion and motivates more adaptive interaction under reliability mismatch (Ma et al., 2021, Burchi et al., 2023). The speech enhancement paper suggests that cross-attention conditioning can extend naturally to vision or multi-speaker settings, though it does not test them (Narayanan et al., 2021). The conversational ASR results suggest that compact latent conversational summaries are more stable than long raw cross-modal histories (Wei et al., 2023). The CSLR paper leaves the exact relative-position formulation of CMRA underspecified, which indicates room for more explicit multimodal positional modeling (Aloysius et al., 2024). OT-based CTC ASR demonstrates large gains on AISHELL-1, but broader multilingual validation and clearer detail on PLM freeze policy and OT hyperparameter sensitivity remain open (Lu et al., 2023).

From these patterns, the most defensible generalization is that Cross-Modal Conformer research revolves around one architectural question: how to preserve the Conformer’s local-global sequence bias while enabling one modality, context stream, or semantic space to alter another in a task-relevant way. The literature currently answers that question with at least four mechanisms—aligned concatenation, modality-axis attention, asymmetric cross-attention, and training-time cross-modal alignment—and treats them not as interchangeable, but as solutions to different structural assumptions about synchronization, reliability, and deployment.