Modality-adaptive Encoder Module (MEM)

• MEM is an encoder-side mechanism that adapts modality-specific representations into features suitable for downstream models.
• It employs varied formulations, including M-Adapter for speech-to-text using pooled self-attention, channel attention in segmentation, and K/V routing in multimodal LLMs.
• MEM bridges modality gaps and reduces computational overhead by compressing long sequences and aligning representation spaces to optimize performance.

A modality-adaptive encoder module (MEM) is an encoder-side mechanism that transforms modality-specific representations into features better matched to a downstream model. In end-to-end speech-to-text translation, the queried notion is instantiated directly by M-Adapter, which is inserted between a pre-trained speech encoder and a pre-trained text decoder to shrink long acoustic sequences and adapt them into text-ready features (Zhao et al., 2022). The acronym is also used in later work for a per-modality enhancement block in semi-supervised multi-modal brain tumor segmentation (Chung et al., 10 Dec 2025) and, in multimodal LLMs, for the interface that connects unimodal encoders to a flexible set of late transformer blocks through learned aligners, gates, and lightweight adapters (Huang et al., 2023). This suggests that MEM is best understood as a functional category defined by modality adaptation at the encoder interface rather than by a single canonical architecture.

1. Terminology and conceptual scope

In the speech translation setting of M-Adapter, the central problem is a modality gap between a pre-trained speech encoder and a pre-trained text decoder. The speech encoder produces long, continuous-valued acoustic representations, whereas the decoder expects shorter, token-level text features. The stated objective of the module is therefore to produce $Z \in \mathbb{R}^{T_t \times d_t}$, with $T_t \ll T_s$ and $d_t$ matched to the decoder’s embedding dimension, from speech features $X \in \mathbb{R}^{T_s \times d_s}$ through a parameterized adapter $A_\theta$, so that the resulting representation is well formed for the decoder (Zhao et al., 2022).

Other works use the same acronym with different local meanings. In semi-supervised brain tumor segmentation, MEM denotes a “Modality-specific Enhancing Module” that applies per-modality channel-wise attention at the encoder stage before complementary fusion (Chung et al., 10 Dec 2025). In multimodal LLMs for embodied AI, a MEM can be viewed as the collection of encoder interfaces, K/V aligners, block selectors, gates, and LoRA-style adapters that enable elastic runtime modality adaptation into the last $N$ LLM blocks (Huang et al., 2023).

Context	MEM meaning	Defining mechanism
Speech-to-text translation	M-Adapter	Strided convolutional pooling embedded in self-attention with residual length matching
Brain tumor segmentation	Modality-specific Enhancing Module	Separate per-modality channel-wise attention with residual enhancement
Multimodal LLMs	MEM as interface components	Encoder token extraction, K/V alignment, late-block gating, and LoRA-based adaptation

A common misconception is to treat MEM as a fixed architecture. The literature instead ties the term to a shared role—encoder-side modality adaptation—while varying the actual operator according to the task: pooled self-attention for speech, channel reweighting for MRI, and latent K/V routing for multimodal LLMs.

2. M-Adapter as a modality-adaptive encoder for speech-to-text translation

M-Adapter is placed between a pre-trained speech encoder and a pre-trained text decoder. In the reported implementation, a lightweight “lit-adapter” composed of two linear layers is inserted before M-Adapter for task-specific adaptation and dimensional alignment; the baselines also use this lit-adapter for fairness (Zhao et al., 2022).

The core design is Multi-head Pooled Self-Attention (MPSA). Each M-Adapter layer replaces standard multi-head self-attention with a pooled variant that models global dependencies through attention while reducing sequence length through 1-D strided convolutions. For an input $X \in \mathbb{R}^{L \times D}$, the layer computes

$Q = XW_Q,\quad K = XW_K,\quad V = XW_V,$

with $W_Q, W_K, W_V \in \mathbb{R}^{D \times D}$. The projected sequences are then pooled: $$Q' = \mathrm{Pool}_Q(Q),\quad K' = \mathrm{Pool}_K(K),\quad V' = \mathrm{Pool}_V(V),$$ where the pooled length is

$T_t \ll T_s$0

Attention is performed on the pooled sequences,

$T_t \ll T_s$1

yielding $T_t \ll T_s$2. Because the attention output has reduced length, the residual path is also pooled: $T_t \ll T_s$3 followed by standard Transformer residual and normalization structure,

$T_t \ll T_s$4

The module is summarized in the paper-consistent operator form

$T_t \ll T_s$5

where $T_t \ll T_s$6 is multi-head attention over pooled $T_t \ll T_s$7, and $T_t \ll T_s$8 is the convolutional pooling applied inside attention and on the residual path. A stack of $T_t \ll T_s$9 such layers shrinks the sequence by $d_t$0. Two evaluated configurations both compress by $d_t$1: mAda1, with $d_t$2, $d_t$3, $d_t$4, $d_t$5; and mAda3, with $d_t$6, each layer using $d_t$7, $d_t$8, $d_t$9.

This architecture addresses two mismatches simultaneously. First, it reduces the discrepancy in sequence length between speech frames and text tokens. Second, it reshapes the representation space so that acoustic outputs become more suitable for a pre-trained text decoder. The design choice to combine convolutional pooling with self-attention is motivated by the observation that purely local CNN compression can lose global dependencies.

3. Interface with pre-trained components and optimization strategy

The speech translation instantiation uses wav2vec 2.0 large as the speech encoder and the mBART decoder as the text decoder. The wav2vec 2.0 encoder is described as pre-trained on 53.2k hours of unlabeled data and fine-tuned on 960 hours of labeled ASR data plus pseudo-labels, while the mBART decoder has 12 layers and is fine-tuned for multilingual MT in the EN$X \in \mathbb{R}^{T_s \times d_s}$050 setting (Zhao et al., 2022).

The adaptation module itself is trained from scratch. The end-to-end objective is standard sequence-to-sequence cross-entropy over target tokens: $X \in \mathbb{R}^{T_s \times d_s}$1 No auxiliary ASR loss, no CTC loss, and no extra data are used in the reported experiments. The training schedule follows a two-step LNA-2step strategy. First, wav2vec 2.0 and mBART are frozen, and only M-Adapter and the lit-adapter are trained. Second, the adapters are fine-tuned together with a subset of wav2vec 2.0 and mBART parameters, specifically layer normalization, encoder self-attention, and encoder–decoder attention. An additional setting trains the entire wav2vec 2.0 encoder while freezing mBART to isolate the contribution of improved speech representations. Training runs for up to 32 epochs with early stopping.

The formal role of the module is thus intermediate rather than terminal. It does not replace either pre-trained endpoint. Instead, it supplies an interface representation that makes the speech encoder and text decoder jointly trainable under the speech translation objective without altering the basic encoder–decoder decomposition.

4. Computational properties and empirical behavior in speech translation

For a Transformer attention layer of length $X \in \mathbb{R}^{T_s \times d_s}$2, the time complexity is $X \in \mathbb{R}^{T_s \times d_s}$3 and the attention matrix requires $X \in \mathbb{R}^{T_s \times d_s}$4 memory. By reducing the speech sequence from $X \in \mathbb{R}^{T_s \times d_s}$5 to $X \in \mathbb{R}^{T_s \times d_s}$6, M-Adapter changes the dominant attention cost from $X \in \mathbb{R}^{T_s \times d_s}$7 to $X \in \mathbb{R}^{T_s \times d_s}$8. With $X \in \mathbb{R}^{T_s \times d_s}$9, the leading attention cost is reduced by approximately $A_\theta$0, while the 1-D convolutional pooling cost $A_\theta$1 remains negligible for long sequences (Zhao et al., 2022).

The reported model sizes show that the adapter variants are larger than the CNN and Transformer baselines, but the paper explicitly notes that the gains are not simply due to parameter count. In the decoder fine-tuned setting, model sizes are 117M for W2V2-cnn-mBART, 147M for W2V2-tran-mBART, 154M for W2V2-mAda1-mBART, and 185M for W2V2-mAda3-mBART. In the decoder frozen setting, the corresponding sizes are 319M, 349M, 357M, and 386M.

On MuST-C EN$A_\theta$2DE, the main results in the decoder fine-tuned setting are 26.12 BLEU for W2V2-cnn-mBART, 26.56 for W2V2-tran-mBART, 27.00 for W2V2-mAda1-mBART, and 27.13 for W2V2-mAda3-mBART. In the decoder frozen, encoder trained setting, the scores are 26.45, 26.91, 27.60, and 27.73, respectively. Across three language pairs, the reported gains over W2V2-cnn-mBART are $A_\theta$3 BLEU on EN$A_\theta$4DE, $A_\theta$5 on EN$A_\theta$6RO, and $A_\theta$7 on EN$A_\theta$8FR, for an average improvement of $A_\theta$9.

The ablations sharpen the interpretation of the module. With $N$0 fixed in W2V2-mAda1-mBART, increasing stride from $N$1 to $N$2 improves BLEU from 26.10 to 27.00, indicating that stronger compression up to $N$3 is beneficial. Pool placement is also decisive: the default MPSA placement outperforms moving pooling before MSA (26.27), before FFN (25.38), or after the second LayerNorm (24.94). When MPSA is disabled at inference time so that only local aggregation remains, BLEU drops from 27.13 to 26.84. The paper further reports that M-Adapter degrades less than the CNN adapter under decreasing resource levels from 408 to 41 hours at mid and high resource, but that both models converge similarly at 41 hours. Together, these results support the interpretation that the module’s value comes from jointly encoding local and global structure while removing sequence redundancy.

5. Alternative MEM formulations in segmentation and multimodal LLMs

In semi-supervised multi-modal brain tumor segmentation, MEM refers to a per-modality enhancement block inserted at the end of each encoder in a dual-branch U-Net. Each encoder has five convolutional stages with channel sizes 32, 64, 128, 256, and 512, with four max-poolings for overall downsampling by $N$4. MEM operates on the deepest feature map $N$5, separately for each modality and without parameter sharing. The mechanism is squeeze-and-excitation-style channel attention: $N$6 Here, $N$7 is ReLU, $N$8 is sigmoid, and no spatial attention is used. The enhanced features are then passed to a Complementary Information Fusion module that concatenates the two modality streams, computes a gated fused representation, and feeds it back to both decoders (Chung et al., 10 Dec 2025).

The segmentation formulation couples MEM to a hybrid semi-supervised objective. For labeled data, each modality uses a weighted sum of Dice and cross-entropy losses with $N$9. For unlabeled data, the model imposes cross-modal prediction consistency through a mean-squared error term, with fixed $X \in \mathbb{R}^{L \times D}$0. On BraTS 2019 HGG, the method is evaluated under 1%, 5%, and 10% labeled fractions. In the T2–T1c ablation at 10% labeled data, the baseline dual-branch U-Net without MEM or CIF reaches DSC 0.6076 and Sens 0.6012; adding MEM only yields DSC 0.6371 and Sens 0.5637; adding CIF only yields DSC 0.7031 and Sens 0.7754; combining MEM and CIF yields DSC 0.7203 and Sens 0.7282. The data therefore distinguish between modality-specific enhancement and cross-modal exchange rather than collapsing them into a single operation.

In multimodal LLMs for embodied AI, the term MEM is used more abstractly. The cited design connects unimodal encoders not to the input layer of a decoder-only LLM, but to a flexible set of the last $X \in \mathbb{R}^{L \times D}$1 transformer blocks. The module comprises unimodal token extraction, two-layer FFN aligners with GELU for K/V projection,

$X \in \mathbb{R}^{L \times D}$2

continuous per-block gates

$X \in \mathbb{R}^{L \times D}$3

and LoRA adapters on K/V projectors in the selected blocks. External multimodal K/V tokens are concatenated with text K/V inside the existing self-attention of each connected block (Huang et al., 2023).

This late-block routing changes the economic profile of adaptation. The reported system achieves up to $X \in \mathbb{R}^{L \times D}$4 reduction in training FLOPs and up to 30% GPU memory reduction relative to fixed-connection baselines while retaining on-par accuracy, and under the same compute budget it improves task accuracy by up to 4%. On nuScenes-QA, one example for OPT-1.3B in the Night C$X \in \mathbb{R}^{L \times D}$5L setting reports 39.2% accuracy, about 1.09 PFLOPs, and 26.0 GB for PromptFuse; 44.7%, about 1.11 PFLOPs, and 27.6 GB for eP-ALM; 46.4%, about 0.87 PFLOPs, and 23.1 GB for mPnP-LLM with $X \in \mathbb{R}^{L \times D}$6; and 46.9%, about 0.93 PFLOPs, and 25.9 GB for $X \in \mathbb{R}^{L \times D}$7. In this formulation, modality adaptation is not sequence compression but selective latent injection into semantically rich LLM blocks.

6. Limitations, interpretive issues, and practical considerations

The speech translation literature imposes a clear boundary on useful compression. The authors cap compression at $X \in \mathbb{R}^{L \times D}$8 because excessive downsampling can make the adapted acoustic sequence shorter than the target text sequence. The same experiments also show that too little compression leaves the mismatch unresolved, while very low-resource conditions reduce the advantage of the adapter (Zhao et al., 2022).

The segmentation formulation introduces different constraints. MEM relies on accurate high-level encoder features, and severe misalignment or missing modalities can reduce its benefit. Under extremely scarce supervision, the per-modality encoders and MEMs can overfit channel reweighting. CIF also assumes the availability of both modalities at inference time, so degraded or absent modalities can lower performance (Chung et al., 10 Dec 2025).

The multimodal LLM formulation inherits yet another set of limitations. It requires labeled, modality-aligned training samples; savings from reducing LLM backprop are bounded when a modality encoder, such as LiDAR, dominates memory use; and simultaneously engaging multiple modalities can sometimes lower accuracy if a modality is uninformative, making gate design important (Huang et al., 2023).

Across these settings, the most stable interpretation is that a MEM is an encoder-side adaptation interface whose form depends on the structure of the modality mismatch. In speech translation, the mismatch is between long acoustic sequences and shorter text-conditioned decoder memory; in MRI segmentation, it is between semantically discrepant imaging sequences; in multimodal LLMs, it is between heterogeneous encoder outputs and the latent spaces of selected transformer blocks. The term therefore denotes a design principle—modality-aware representation adaptation before downstream fusion or decoding—rather than a single universally adopted module.