Decoupled Cross-Attention Transfer (D-CAT)

• The paper introduces a decoupled framework where cross-attention is used only during training to align a frozen source modality with an updated target modality.
• It employs a masked alignment loss alongside a classification objective, achieving up to 10% F1-score gains in in-distribution settings.
• Empirical results demonstrate enhanced unimodal inference from paired multimodal training, reducing hardware redundancy and deployment costs.

Decoupled Cross-Attention Transfer (D-CAT) is a cross-modal transfer learning framework that aligns modality-specific feature spaces during training while preserving independent unimodal inference pipelines. It was introduced to address a recurring deployment mismatch in multimodal systems such as human activity recognition in human-robot collaboration: paired sensor suites such as video, audio, and IMU may be available during data collection and training, whereas real deployments often operate with only a subset of sensors because full sensor suites are not economically or technically feasible. D-CAT therefore uses paired modalities during training, but requires only a single target modality at test time. The method pretrains and freezes a source-modality network, trains a target-modality network with a standard classification loss plus a masked cross-attention alignment loss, and does not use the source pathway during inference (Daher et al., 11 Sep 2025).

1. Problem formulation and conceptual position

D-CAT is defined by a decoupling principle: classification pipelines remain modality-specific, and cross-modal transfer occurs only through an auxiliary alignment mechanism during training. This distinguishes it from multimodal fusion approaches that couple modalities through joint attention or fusion and therefore require all modalities at both training and inference. In D-CAT, the source modality network is pretrained and frozen; the target modality network is updated so that its self-attention-derived representation aligns with the frozen source representation, while still optimizing its own classification objective (Daher et al., 11 Sep 2025).

The practical motivation is explicit. Rich paired sensor suites can be used during development, but perception systems deployed in cost-sensitive or adaptive environments may have to function with only one sensor. D-CAT targets that regime by transferring knowledge from a source modality to a target modality without introducing a test-time dependency on the source modality. In the paper’s framing, this is intended for settings such as assistive robots in homes with variable sensor availability (Daher et al., 11 Sep 2025).

A common misconception is to treat D-CAT as a standard multimodal fusion model. It is not. The cross-attention mechanism is used only to define an alignment loss during training, and the final deployed model is unimodal. Another misconception is that source–target coupling persists at inference. The method is explicitly designed so that the cross-attention module and source modality are not used at test time (Daher et al., 11 Sep 2025).

2. Architecture and modality-specific encoders

The architecture consists of modality-specific encoders, modality-specific self-attention modules, decoupled classifier heads, and a training-only cross-attention alignment module. Each encoder outputs a sequence of embeddings $E \in \mathbb{R}^{T_m \times d_{\text{model}}}$. A self-attention block for each modality maps $E$ to queries, keys, and values and produces contextualized features for classification. The same $Q$, $K$, and $V$ tensors are also exposed to the D-CAT loss. The source encoder and source self-attention module are frozen during transfer; only the target encoder, target self-attention module, and target classifier are updated (Daher et al., 11 Sep 2025).

The classifier heads are explicitly decoupled. Each modality has its own head downstream of its self-attention module, and no fusion is performed at inference. During training on a paired sample $(x^{(A)}, x^{(B)})$, the source pathway supplies frozen $K_A$ and $V_A$, while the target pathway supplies trainable $K_B$ and $V_B$. Classification loss is computed on the target modality only, and the alignment loss is backpropagated through the target network only (Daher et al., 11 Sep 2025).

The modality-specific preprocessing and encoder choices are fixed as follows.

Modality	Preprocessing	Encoder
IMU	min–max normalize to $E$0	1D CNN backbone inspired by Samosa
Video	resize each frame to $E$1	2D CNN backbone (Image ResNet-101)
Audio	STFT $E$2 Mel filter bank $E$3 log-Mel spectrogram	PANNs backbone

For IMU, the encoder is a 1D CNN backbone inspired by Samosa and adapted to datasets; convolution blocks include Conv $E$4 BatchNorm $E$5 ReLU, with pooling and dropout stages, and the block diagram comprises an initial convolution block plus pool, then two additional convolution blocks, followed by final pool and dropout. For video, the encoder is an Image ResNet-101 in which a residual block is repeated 33 times, combining pairs of 2D convolutions with batch normalization and ReLU. For audio, the encoder is based on PANNs, with two consecutive convolutional blocks, followed by pooling plus dropout repeated six times, and a fully connected layer plus ReLU plus dropout (Daher et al., 11 Sep 2025).

3. Attention formulation and optimization objective

Let $E$6 denote the embedding matrix of a sequence of length $E$7 produced by a modality-specific encoder. D-CAT defines the key, query, and value projections by linear layers: $E$8

$E$9

$Q$0

The self-attention output is

$Q$1

which produces a sequence of the same length, where each element is a weighted sum of all elements in the original sequence. Cross-attention from source modality $Q$2 to target modality $Q$3 is defined as

$Q$4

Here $Q$5 is the target query matrix, while $Q$6 and $Q$7 come from the frozen source modality (Daher et al., 11 Sep 2025).

The paper’s alignment objective is to enforce

$Q$8

Using the Frobenius norm

$Q$9

the cross-attention loss with normalization is written as

$K$0

The paper then factors out $K$1 and drops softmax and scaling, yielding the final alignment loss

$K$2

The stated rationale is that, by the theorem provided in the paper, minimizing this loss drives $K$3 and $K$4 to linear mappings of $K$5 and $K$6, respectively, ensuring alignment even without explicit softmax and scaling (Daher et al., 11 Sep 2025).

To reduce negative transfer from source errors, D-CAT applies masked cross-modal alignment. An indicator function $K$7 is 1 if the source prediction on sample $K$8 is correct and 0 otherwise, and the final masked loss is

$K$9

Classification uses cross-entropy

$V$0

and the total objective is

$V$1

where $V$2 controls the balance between classification and alignment. Optimization is performed over the target encoder, target self-attention, and target classifier, while the source network remains frozen (Daher et al., 11 Sep 2025).

4. Training protocol, synchronization, and evaluation setup

D-CAT requires paired, synchronized modalities during training. Each dataset provides synchronized modality streams, which are windowed to fixed lengths per modality. The paper reports the following window or stride lengths: UESTC-IMU 70, UESTC-Video 30, Cough-IMU 60, Cough-Audio 0.5 seconds, VGGSound-Video 30, and VGGSound-Audio 2 seconds. Batches contain paired samples for source and target modalities so that $V$3 can be computed, while $V$4 is computed only on the target outputs (Daher et al., 11 Sep 2025).

The reported optimization setup uses Adam, attention size 512, and ReLU activations. Dropout and weight decay are tuned via grid search. Dataset-specific settings are: UESTC-IMU epochs 10, learning rate $V$5, dropout 0.8, batch 16, weight decay 0.005; UESTC-Video epochs 100, learning rate $V$6, dropout 0.8, batch 8, weight decay 0.005; Cough-IMU epochs 10, learning rate $V$7, dropout 0.8, batch 32, weight decay 0.001; Cough-Audio epochs 100, learning rate $V$8, dropout 0.6, batch 32, weight decay 0.0001; VGGSound-Video epochs 100, learning rate $V$9, dropout 0.5, batch 8, weight decay 0.005; VGGSound-Audio epochs 150, learning rate $(x^{(A)}, x^{(B)})$0, dropout 0.5, batch 8, weight decay 0.0. The reported seeds are UESTC-IMU 11, UESTC-Video 17, Cough-IMU 7, Cough-Audio 17, VGGSound-Video 17, and VGGSound-Audio 11 (Daher et al., 11 Sep 2025).

The evaluation covers three multimodal datasets.

Dataset	Modalities and size	Split
UESTC-MMEA-CL	Video-IMU, 6,522 samples	80% train, 10% val, 10% test
Cough Audio-IMU	Audio-IMU, 2,576 samples	65% train, 18% val, 17% test
VGGSound	Audio-Video, 2,886 clips	70% train, 15% val, 15% test

UESTC-MMEA-CL contains 10 participants and 32 activities, but the experiments use a subset of 8 classes: drinking, reading, floor sweeping, cutting fruits, washing hands, typing on a laptop, typing on a phone, and opening/closing a door. Cough Audio-IMU contains 13 participants and 8 tasks; the dataset is imbalanced, and minimal-motion IMU samples were removed. VGGSound contains 309 classes, but the experiments use a subset of 9 classes. The paper distinguishes in-distribution (subjects may appear across train/val/test) from out-of-distribution (no subject overlap across splits); for VGGSound, user annotations are unavailable, and the paper evaluates OOD on VGGSound under its protocol (Daher et al., 11 Sep 2025).

At inference, only the target modality’s encoder, self-attention, and classifier are used. No source network, cross-attention computation, or paired data are required. This eliminates the need to deploy multiple sensors and run multimodal fusion models at inference, and reduces hardware redundancy as well as compute and memory cost (Daher et al., 11 Sep 2025).

5. Empirical behavior in in-distribution and out-of-distribution regimes

In in-distribution settings, D-CAT is most effective when transferring from a stronger modality to a weaker modality. The paper reports up to roughly $(x^{(A)}, x^{(B)})$1 F1-score gains overall over uni-modal training. On UESTC-IMU, baseline F1 improves from 0.893 to 0.967, accuracy from 0.877 to 0.969, recall from 0.895 to 0.967, and precision from 0.900 to 0.974. On Cough-IMU, baseline F1 improves from 0.205 to 0.256, accuracy from 0.440 to 0.474, recall from 0.268 to 0.304, and precision from 0.185 to 0.243. By contrast, transferring from weaker IMU to stronger modalities in ID typically does not help and can slightly reduce performance, as noted for UESTC-Image and Cough-Audio (Daher et al., 11 Sep 2025).

In out-of-distribution settings, the transfer pattern changes. The paper reports that even weaker source modalities can help stronger target modalities under distribution shift, provided the target model is not overfitted on the training data. On UESTC-Image, IMU$(x^{(A)}, x^{(B)})$2Image increases accuracy from 0.518 to 0.601, recall from 0.518 to 0.595, precision from 0.540 to 0.627, and F1 from 0.508 to 0.597. On Cough-Audio, IMU$(x^{(A)}, x^{(B)})$3Audio increases accuracy from 0.607 to 0.680, recall from 0.533 to 0.618, precision from 0.477 to 0.572, and F1 from 0.482 to 0.578. On VGGSound-Audio, Image$(x^{(A)}, x^{(B)})$4Audio increases accuracy from 0.897 to 0.926, recall from 0.828 to 0.883, precision from 0.889 to 0.909, and F1 from 0.839 to 0.892 (Daher et al., 11 Sep 2025).

The paper attributes reduced OOD gains in some cases to target-model overfitting, particularly for IMU and Image on UESTC. The stated hypothesis is that overfitted models struggle to generalize to unseen subjects, which reduces the effectiveness of alignment. This interpretation also structures the paper’s practical advice: transfer directionality should be validated per dataset, especially under OOD conditions (Daher et al., 11 Sep 2025).

Ablation studies further clarify the method’s behavior. Masked cross-modal alignment (MCMA) consistently improves over no MCMA in ID when transfer is effective, but in OOD no MCMA sometimes slightly outperforms MCMA, as in Cough-Audio and VGGSound-Audio, with differences reported as small ($(x^{(A)}, x^{(B)})$5 absolute). A $(x^{(A)}, x^{(B)})$6 ablation over $(x^{(A)}, x^{(B)})$7 generally favors $(x^{(A)}, x^{(B)})$8; performance degrades when alignment overwhelms classification, with an example given for UESTC-IMU in which accuracy is 0.470 at $(x^{(A)}, x^{(B)})$9 and 0.321 at $K_A$0. The baselines are a Samosa-like 1D CNN for IMU, ResNet-101 for image, and PANNs for audio. The paper does not include quantitative comparisons to other transfer methods such as KD, MMD, or CCA/CORAL, although it discusses them in related work (Daher et al., 11 Sep 2025).

6. Limitations, failure modes, and implementation guidance

The method has several explicit limitations. It requires paired, time-synchronized modalities during training. Negative transfer is possible if source classifications are noisy, and masked alignment mitigates but does not eliminate that risk. Sensitivity to overfitting in the target model can reduce OOD transfer gains. The theoretical justification is linear in $K_A$1 space, so highly nonlinear cross-modal relationships may not be fully captured. In addition, the method drops softmax and scaling in the alignment term based on the paper’s theorem; the paper notes that alternative normalization or explicit attention alignment might be beneficial in some settings (Daher et al., 11 Sep 2025).

The deployment advantages follow directly from the decoupling. Because only the target pipeline is used at inference, D-CAT reduces hardware redundancy and runtime relative to coupled multimodal fusion, and it is described as suitable for cost-sensitive or adaptive environments such as assistive robots where cameras may not be available. This suggests a design pattern in which multimodal supervision is used during training, but the deployed system remains unimodal (Daher et al., 11 Sep 2025).

The reported practical guidance is correspondingly specific. For ID, a stronger source such as video or audio should be selected to improve a weaker target such as IMU. For OOD, even weaker sources can help stronger targets if the target is not overfitted, but transfer directionality should be validated per dataset. The recommended starting point is $K_A$2, with tuning in $K_A$3 and avoidance of very large $K_A$4. MCMA is recommended in ID; in OOD, both MCMA and no-MCMA should be tested and selected based on validation. Stronger regularization, early stopping, robust data augmentation, cross-validation across subjects, and stratified splits are recommended to reduce overfitting. The source modality should be pretrained to good accuracy and then frozen, and synchronization quality between modalities should be maintained during training. The implementation is available in the repository released by the authors, with dataset-specific configurations and reported seeds for reproducibility (Daher et al., 11 Sep 2025).

The paper also outlines extensions rather than evaluated variants. These include multi-source alignment

$K_A$5

handling asynchronous sampling rates through interpolation, resampling, or learned temporal alignment, multi-head cross-attention alignment through per-head or aggregated losses, and combining D-CAT with adversarial feature alignment or contrastive objectives for domain adaptation. These are presented as extensions rather than empirical claims (Daher et al., 11 Sep 2025).

7. Related use of decoupled cross-attention transfer in machine translation

A related but separately framed use of decoupled cross-attention transfer appears in machine translation. The paper "Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation" does not name its method D-CAT, but its $K_A$6 setting is described as the same core idea: decouple cross-attention from the rest of the model and transfer only that submodule while freezing the encoder and decoder bodies. In that setting, the trainable parameters are the decoder cross-attention weights $K_A$7 and the cross-attention layer-normalization parameters, together with newly initialized embeddings for the new language side (Gheini et al., 2021).

The MT formulation is architecturally different from the sensor-modality setting, but the structural analogy is direct. A pretrained parent Transformer-base in fairseq with 6 encoder layers, 6 decoder layers, $K_A$8, and $K_A$9 heads is adapted to new language pairs by updating only cross-attention and new-side embeddings. Across six child pairs, the reported parameter fractions are: scratch 100%, embeddings only 8%, full-body fine-tuning 75%, and $V_A$0 17%. The paper reports that this cross-attention-only strategy is nearly as effective as full fine-tuning, with pairwise drops of about 0.1 to 2.0 BLEU depending on the language pair, while also producing substantial storage savings and mitigating catastrophic forgetting relative to full-body adaptation (Gheini et al., 2021).

The MT results also emphasize a representational effect that is conceptually relevant to D-CAT more generally. Freezing encoder and decoder bodies forces new-side embeddings to align to the parent representation space, and updating cross-attention learns the bridge between the frozen spaces. In bilingual lexicon induction, the paper reports markedly higher alignment accuracies under the cross-attention-only setting than under full fine-tuning, such as 55% versus 19.7% for De$V_A$1Fr induction on a strict exact-match metric. This does not establish identity between the MT and sensor-modality formulations, but it does show that decoupled cross-attention transfer is a broader transfer-learning pattern rather than a domain-specific heuristic (Gheini et al., 2021).