CTAM: Cross-Task Attention Module for MTL

• CTAM is a neural architectural component for multi-task learning that facilitates dynamic integration of task-specific features.
• It leverages various attention mechanisms—including additive, dot-product, self-attention, and correlation-guided variants—to selectively fuse information across tasks.
• Empirical studies demonstrate that CTAM improves prediction accuracy and interpretability while incurring modest computational overhead.

A Cross-Task Attention Module (CTAM) is a neural architectural component designed for multi-task learning (MTL) scenarios. Its primary function is to explicitly facilitate information exchange and integration across different task-specific or attribute-specific representations within a single network. By leveraging various forms of attention mechanisms—including additive, multiplicative (dot-product), self-attention, and correlation-guided variants—CTAMs enable the dynamic re-weighting, fusion, or gating of features between interconnected tasks, thereby enhancing both overall prediction performance and interpretability of task relationships.

1. Conceptual Foundations of Cross-Task Attention

The central motivation for CTAMs is that supervised multi-task networks often process several related outputs (tasks or attributes) that are non-independent but manifest complex interrelations. Conventional MTL models—employing only hard parameter sharing or unstructured feature concatenation—fail to capture these dependencies or permit selective, context-aware feature sharing.

CTAMs generalize the “query–key–value” attention paradigm to the multi-task setting, enabling the feature stream for one task (the target) to “attend” over the features of other tasks (sources). This allows the model to learn which auxiliary signals are relevant for refining its prediction at each spatial location or channel, and to suppress unrelated or detrimental information. Cross-task attention can be implemented at various network depths, from early shared encoders (Kim et al., 2023) to late fusion in task-specific decoders (Lopes et al., 2022, Udugama et al., 20 Oct 2025, Kim et al., 2022), and via element-wise, channel-wise, or spatial mechanisms.

2. Mathematical Formalisms of CTAM Across Architectures

A broad spectrum of CTAM instantiations exists, differing in implementation specifics while maintaining a common logic:

1. Matrix Fusion via Learnable MLP (Attribute-level): In the attribute-based scoring of lung nodules (Fu et al., 2021), the CTAM (there termed CAAM) aggregates K attribute-specific vectors $f_s^{(i)} \in \mathbb{R}^{1\times d}$ into a matrix $X_s \in \mathbb{R}^{K \times d}$. Each target attribute $t$ is assigned a two-layer MLP generating a softmax-normalized attention vector $a^{(t)} \in \mathbb{R}^K$ over attributes:

$$\widetilde f^{(t)} = \sum_{i=1}^K a_i^{(t)} f_s^{(i)}$$

These attended vectors are then regressed to obtain per-attribute predictions.

2. Query–Key–Value Cross Attention (Pixel-level, Task-wise): In pixel/dense prediction settings (Kim et al., 2023, Udugama et al., 20 Oct 2025, Kim et al., 2022), task-specific features are flattened or windowed, and cross-task aggregation is performed as:

$$Q = l_q(F^{(i)}) \in \mathbb{R}^{L\times d_k}, \quad K = l_k(S) \in \mathbb{R}^{L\times d_k}, \quad V = l_v(S) \in \mathbb{R}^{L\times d_v}$$

$$A = \mathrm{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right)V$$

The attended output $A$ is concatenated with the original target and fused via a $1\times1$ convolution.

1. Correlation-Guided Bidirectional Fusion: In models like DenseMTL (Lopes et al., 2022), for each pair of tasks $(i,j)$, CTAM computes a downsampled, projected correlation matrix to weight one task’s features by spatial similarity with another, supplemented by a self-attention branch for private cues. The outputs are concatenated with learnable per-channel weights and reintegrated via a $1\times1$ convolution.
2. Additive (Bahdanau-style) Attention (Vector-level): In affective behavior analysis (Nguyen et al., 2022), vector representations from two tasks (e.g., action unit detection and facial expression recognition) are projected and combined to modulate the output logits of one branch as a function of the other.
3. Guided Channel Attention Using Upstream Masking: In sequential or cascaded setups (Zhou et al., 2019), CTAM (often termed cross-task guided attention, CGA) leverages the previous task’s soft segmentation predictions to modulate channel importance on the next task’s feature maps, recalibrating responses toward context-relevant regions.
4. Non-local Attention with Task Memory Aggregation: In temporal and tracking applications (Guo et al., 2021), CTAM computes attention maps between the current state of one task (e.g., target embedding) and the historical memory or embedding of another, generating temporally-aware features that facilitate robust fusion.

3. Integration in Multi-Task Network Architectures

CTAMs are generally placed at modular interfaces between shared backbones and task-specific heads. Common architectural strategies include:

• Encoder-level CTAM: Inserted after shared encoder blocks, so that per-task features at intermediate scales are refined jointly via cross-attention before downstream decoding (Kim et al., 2023, Kim et al., 2022).
• Decoder-level or Late-fusion CTAM: Placed in dedicated multi-task exchange blocks or in fusion modules just prior to the final prediction head (Udugama et al., 20 Oct 2025, Lopes et al., 2022).
• Attribute-level CTAM: For multi-attribute scoring, a CTAM operates after the per-attribute modulation, allowing explicit attribute-to-attribute interaction (Fu et al., 2021).
• Temporal and Graph-based CTAM: Leveraging CTAM with spatio-temporal memory (Guo et al., 2021) or graph neural networks (Nguyen et al., 2022) to enrich inter-task context.

The precise integration point is typically determined by the spatial or semantic abstraction at which cross-task dependencies are hypothesized to be most salient.

4. Hyperparameters, Computational Overhead, and Training

CTAM design involves several hyperparameters: projection dimensions (e.g., $d_k, d_v$), number of attention heads, location and frequency of module insertion, window sizes for local attention, and per-channel weighting factors.

The computational/load overhead introduced by CTAM is modest in modern settings. For example, CTAN’s CTAE and CTAB add $1$–$2$M parameters (∼8%) to a ResNet-50+decoders model and increase compute by $\sim10\%$ (Kim et al., 2023). Window-based CTAMs split attention into local groups, reducing the cost from $\mathcal{O}(HW^2C)$ to $\mathcal{O}(HWp^2C/h)$ where $p$ is window size, $h$ is the number of heads (Udugama et al., 20 Oct 2025).

CTAMs are typically trained end-to-end with standard task losses (e.g., cross-entropy for segmentation/classification, mean-squared error for regression), often with dynamic loss weighting across tasks (such as Dynamic Weight Averaging). No explicit losses are needed for CTAM parameters, as gradients flow from downstream task heads through the CTAM blocks.

5. Empirical Impact and Interpretability

A consistent finding across domains is that the use of CTAM yields significant gains over both naïve MTL baselines (hard parameter sharing) and per-task attention (MTAN):

• In multi-attribute CT for lung nodules, CTAM reduced MAE from 0.455 to 0.405, further dropping to 0.398 when combined with all attention modules (Fu et al., 2021).
• In CTAN for medical imaging, CTAM led to 3–5% improvement in segmentation, diagnosis, and dose estimation tasks over single-task and MTAN baselines (Kim et al., 2023).
• Window-based CTAM in M2H improved semantic segmentation mIoU by 4–5 points, depth RMSE by >10%, and boundary detection by ∼6 points versus previous SOTA models (Udugama et al., 20 Oct 2025).
• CTAMs provide interpretable attention weights corresponding to clinically or semantically meaningful interdependencies (e.g., strong attention from “spiculation” to “malignancy” mirrors known radiological relationships (Fu et al., 2021)).
• In robust tracking and behavioral analysis, CTAMs enabled higher accuracy and robustness under occlusion and ambiguous visual signals (Guo et al., 2021, Nguyen et al., 2022).

These improvements are often supported by ablation studies isolating the effect of CTAM from other architectural changes.

6. Representative Variants and Their Applications

The table below summarizes core CTAM variants and their respective implementations:

Paper/Framework	CTAM Variant	Integration Point
(Fu et al., 2021) (Lung Nodules)	Two-layer MLP attention (CAAM)	Attribute-level post-modulation
(Kim et al., 2023) (CTAN)	QKV cross-attention, encoder/bottleneck	Encoder, bottleneck
(Udugama et al., 20 Oct 2025) (M2H)	Window-Based Multi-Head Attention	Decoder, pre-task-head
(Lopes et al., 2022) (DenseMTL)	Correlation-guided & self-attention	Decoder, multi-task exchange
(Kim et al., 2022) (SCAMTL)	Query–key–value (per-scale)	All scales, pre cross-scale
(Guo et al., 2021) (TADAM, MOT)	Temporal-aware, non-local attention	Position & embedding branches
(Zhou et al., 2019) (OM-Net)	Cross-task guided channel attention	Inside each task branch
(Nguyen et al., 2022) (Affective MTL)	Additive (Bahdanau) vector attention	Expression gating, head-to-head

Each variant is algorithmically optimized for the information structure (spatial, attribute, temporal) and computational regime (dense windows, cascades, graphs) of its target application.

7. Limitations and Future Directions

Reported limitations include:

• Lack of temporal consistency in video processing unless specifically addressed (Udugama et al., 20 Oct 2025).
• Sensitivity to alignment of spatial feature maps across tasks.
• Degradation of attention signal for fine-grained structures smaller than the attention window.
• In some cases, additional parameter and compute cost may restrict deployment for very lightweight or real-time applications.
• The need for losses that adequately incentivize effective information sharing without negative transfer.

Future research could explore hierarchical CTAMs spanning multiple scales and tasks, adaptive gating mechanisms, integration with domain adaptation pipelines, and extending CTAMs for recurrent and transformer-based backbones. Applications of CTAM in unsupervised multi-task adaptation and open-vocabulary multi-label scenarios constitute additional promising directions.

---

For the complete methodological and empirical details for each described CTAM instantiation, refer to (Fu et al., 2021, Kim et al., 2023, Udugama et al., 20 Oct 2025, Lopes et al., 2022, Kim et al., 2022, Nguyen et al., 2022, Zhou et al., 2019), and (Guo et al., 2021).