Cross-Modality Focus Fusion

• Cross-Modality Focus Fusion (CMFF) is a multimodal strategy that employs selective attention to emphasize useful information while suppressing redundant or interfering signals.
• It integrates diverse mechanisms such as modality-level weighting, mask-guided feature fusion, and channel-spatial enhancement to optimize performance in tasks like segmentation, detection, and survival prediction.
• Empirical results from applications in medical imaging, RGB–IR detection, and multimodal registration demonstrate that CMFF significantly improves accuracy and robustness compared to traditional fusion methods.

Cross-Modality Focus Fusion (CMFF) denotes a class of multimodal learning strategies in which cross-modal interaction is made selective rather than indiscriminate. In these systems, fusion is driven by learned emphasis over channels, pixels, parts, tokens, latent states, regions of interest, or modality-level weights, so that complementary information is amplified while interference, redundancy, or degradation is suppressed. Taken together, the literature suggests that CMFF is not a single canonical block but a family of task-oriented mechanisms spanning medical segmentation, survival prediction, RGB–infrared and RGB–event detection, multimodal LLMs, optical/scene flow, and multimodal image registration (Zhang et al., 2022, Chen et al., 2024, Yang et al., 4 Sep 2025).

1. Terminology, scope, and historical formation

An early precursor appears in attention-based multimodal fusion for affect analysis, where hierarchical delta-attention was proposed to preserve long-range dependencies within and across modalities, while using delta-attention to focus on local differences per modality and cross-attention fusion to obtain a global view of the emotion expressed through these delta-self-attended modalities. That work also stated that the addition of attention was new to the multi-modal fusion field and was being scrutinized with respect to the stage at which attention should be used (Panchal, 2020).

Subsequent work diversified both the meaning and the implementation of focused fusion. In RGB–infrared person re-identification, Dual-level Feature Fusion and Affinity Modeling combined learned attention from local to global manner with supervised intra- and inter-modality affinities (Yin et al., 2021). In brain tumor segmentation, CMFF was explicitly defined as a cross-modality feature fusion process paired with cross-modality feature transition, with a dedicated fusion branch and mask-guided feature learning (Zhang et al., 2022). In non-small cell lung cancer survival prediction, cross-modality attention-based multimodal fusion was realized as a modality-wise attention pooling block over pathology and RNA-seq embeddings (Deng et al., 2023). In vision–language modeling, a pipeline-wide integration paradigm combined Text-Guided Unified Vision Encoding, Context-Aware Recursive Alignment Decoding, and Dual-Supervised Semantic Mapping Loss to achieve deep, dynamic integration throughout the processing pipeline (Liu et al., 14 Apr 2025). In multimodal flow estimation and registration, unified latent spaces and state-space duality introduced reliability-aware or structure-aware fusion operators anchored in shared representations rather than only in raw feature concatenation (Guo et al., 17 Mar 2026, Li et al., 2 Jun 2026).

Setting	CMFF realization	Representative paper
Brain tumor segmentation	mask-guided feature fusion branch after modality-specific segmentation branches	(Zhang et al., 2022)
Survival prediction	modality-wise softmax attention over image and RNA-seq embeddings	(Deng et al., 2023)
RGB–IR detection	dual semantic/spatial enhancement in feature space	(Chen et al., 2024)
RGB–event detection	EGMS-guided sparse token fusion with CAE and FI-Mamba	(Yang et al., 4 Sep 2025)
Vision–language MLLM	text-guided encoder fusion plus recursive alignment decoding	(Liu et al., 14 Apr 2025)
Multimodal flow	Event Edge Space with reliability-aware adaptive fusion	(Guo et al., 17 Mar 2026)
Multimodal registration	cross-form SSD via CMI and progressive MSF	(Li et al., 2 Jun 2026)

2. Core mechanisms of focus

A first recurrent form of CMFF is modality-level weighting. In multimodal survival prediction, the cross-modality attention-based multimodal fusion block operates on modality embeddings $f_m \in \mathbb{R}^{L}$, applies two shared $1 \times 1$ convolutional layers with Tanh activation, computes scalar modality scores, normalizes them by softmax,

$$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$

and forms the fused representation

$F = \sum_{m=1}^{M} a_m f_m.$

The paper explicitly noted that this is not Transformer query–key–value attention; the cross-modality aspect arises from shared parameters and softmax coupling across modalities, which makes each modality’s importance relative to the others for a given sample (Deng et al., 2023).

A second form is feature-level fusion guided by task predictions. In brain tumor segmentation, the CMFF network contains two single-modality feature learning branches $S_A$ and $S_B$ and a cross-modality feature fusion branch $S_F$. The deep features in the last two convolutional layers, as well as the outputs of the single-modality branches, are connected to the fusion branch. The distinguishing component is a mask-guided feature learning scheme in which the segmentation masks predicted by $S_A$ and $S_B$ serve as attention masks for the fused features. The total objective jointly optimizes all three branches,

$$\arg \min_{S_A,S_B,S_F} \mathcal{L}_{seg}(S_A)+\mathcal{L}_{seg}(S_B)+\mathcal{L}_{seg}(S_F),$$

with Dice loss for each branch (Zhang et al., 2022).

A third form is channel- and spatial-domain cross-modality emphasis. In RGB–infrared detection, DEYOLO fuses RGB–IR in feature space rather than image space and makes the fusion detection-centered. The DECA module computes a mixed cross-modality feature, derives shared channel weights $1 \times 1$0, and uses softmax across channels to refine modality-specific channel weights. The DEPA module analogously constructs mixed spatial weights $1 \times 1$1, applies softmax over spatial positions, and cross-applies the enhanced weights to the opposite modality. The resulting design was explicitly described as a dual-enhancement mechanism: fusion-to-modality enhancement and modality-to-modality enhancement, both aimed at focusing on complementary, detection-relevant information while suppressing mutual interference (Chen et al., 2024).

A related Mamba-based image-fusion formulation uses Dynamic Feature Fusion Module, combining Dynamic Feature Enhancement Module and Cross Modality Fusion Mamba Module. There the fusion path mixes modalities multiplicatively and additively, runs a shared ES2D Mamba operator for global correlation modeling, and then re-weights channels by Efficient Channel Attention. This suggests a CMFF interpretation in which focus can be realized without explicit softmax attention, through difference perception, state-space propagation, and channel reweighting (Xie et al., 2024).

3. Sparse, state-space, and latent-space CMFF

Recent work extends CMFF from dense feature maps to sparse tokens and hidden-state interactions. In RGB–event detection, FocusMamba first performs Event-Guided Multimodal Sparsification, producing binary sparsification maps $1 \times 1$2 and $1 \times 1$3. Complementarity-Aware Enhancement then boosts regions where one modality is kept and the other is not, while Focused Interlaced Mamba constructs the union mask $1 \times 1$4, gathers only important tokens, interleaves RGB and event sequences, and applies Bidi-Scan Mamba only to those selected tokens. In this formulation, “focus” is literal token selection plus mask-based complementary enhancement, not merely a reweighting over dense grids (Yang et al., 4 Sep 2025).

A second state-space realization appears in RGB–IR object detection with Fusion-Mamba. Its Fusion-Mamba block combines State Space Channel Swapping for shallow fusion and Dual State Space Fusion for deep fusion. Features from each modality are first mixed at the channel level, then mapped into a hidden state space via SS2D/VSS, and finally coupled by dual gated interactions,

$1 \times 1$5

with the symmetric equation for the infrared branch. The stated objective is to reduce disparities between cross-modal features and enhance the representation consistency of fused features by associating cross-modal features in a hidden state space based on an improved Mamba with a gating mechanism (Dong et al., 2024).

Multimodal registration provides a more explicitly structural variant. RegNetMamba-2 uses Structured State Space Duality in a coarse-to-fine network, introducing a Cross-Modality feature Interaction module and a Multi-Scale feature Fusion module. Its local-enhanced SSD uses a feature scaling function

$1 \times 1$6

with $1 \times 1$7, to sharpen similarities so that highly correlated tokens are emphasized and weakly correlated ones are suppressed. Cross-form SSD then lets one modality extract global structure from the other without joint concatenation, while progressive upward fusion propagates cross-modal information from $1 \times 1$8 to $1 \times 1$9 scale (Li et al., 2 Jun 2026).

A broader latent-space formulation appears in multimodal flow estimation. There, event-derived spatiotemporal edges define an Event Edge Space, and image and LiDAR features are explicitly aligned to this shared representation. Fusion is then reliability-aware: $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$0 where $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$1 is a global reliability score and $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$2 is local spatial reliability. This formulation makes focus depend on estimated modality trustworthiness rather than only on correlation magnitude (Guo et al., 17 Mar 2026).

4. Domain-specific realizations

In medical imaging, CMFF primarily serves complementarity between heterogeneous evidence sources. For brain tumor segmentation, the complementary sources are MRI modality pairs grouped as $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$3 and $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$4, with the fusion branch learning to combine deep features and predicted masks from both branches (Zhang et al., 2022). For non-small cell lung cancer survival prediction, CMFF operates at a coarser semantic level, using a modality-wise attention pooling over patient-level histopathology and RNA-seq embeddings to determine sample-specific reliance on tissue or transcriptomic features (Deng et al., 2023). In RGB–infrared person re-identification, a related design fuses part-level and global features and augments them with affinity modeling so that feature fusion and pairwise relations are both modality-aware (Yin et al., 2021).

In object detection, the dominant shift is from undifferentiated fusion to detection-centered or foreground-centered fusion. DEYOLO locates fusion between backbone and neck/head and explicitly rejects image-fusion-first pipelines in favor of feature-space aggregation through DECA and DEPA (Chen et al., 2024). SFEDet pushes this idea further: two lightweight YOLOv8-S RPNs generate high-recall proposals independently in RGB and thermal, their union is filtered by NMS, and only those sparse RoIs are sent to a heavy Fusion-driven Examination and Refinement stage. This suggests a CMFF pattern in which the primary selection variable is not channel or pixel weight but proposal membership itself (Tian et al., 29 Jun 2026).

In multimodal LLMs, CMFF becomes pipeline-wide integration. FUSION conditions the vision encoder on text through Text-Guided Unified Vision Encoding, then recursively updates latent visual tokens during decoding through Context-Aware Recursive Alignment Decoding, and finally regularizes both vision-to-text and text-to-vision mappings by Dual-Supervised Semantic Mapping Loss. The effect is a cross-modality focusing mechanism at encoder, decoder, and representation-alignment levels rather than at a single fusion layer (Liu et al., 14 Apr 2025).

5. Empirical behavior and quantitative evidence

Across domains, focused fusion typically improves both accuracy and robustness over naive or dense baselines. In brain tumor segmentation, the full CMFF model on BraTS 2018 achieved average Dice $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$5, compared with $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$6 for the stronger single-modality branch $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$7, $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$8 for $$a_m = \frac{\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)}{\sum_{m=1}^{M}\exp\left(\mathbf{W}^\mathrm{T}\,\text{Tanh}(\mathbf{V}f_m^\mathrm{T})\right)},$$9, and $F = \sum_{m=1}^{M} a_m f_m.$0 for simple output averaging $F = \sum_{m=1}^{M} a_m f_m.$1. On BraTS 2017, the full model reached average Dice $F = \sum_{m=1}^{M} a_m f_m.$2, while the variant without mask-guided fusion scored $F = \sum_{m=1}^{M} a_m f_m.$3, indicating that feature-level fusion itself is beneficial and that mask-guided attention adds further gain (Zhang et al., 2022).

In multimodal survival prediction, the proposed cross-modality attention fusion reached c-index $F = \sum_{m=1}^{M} a_m f_m.$4, whereas single-modality learning achieved $F = \sum_{m=1}^{M} a_m f_m.$5 with solely tissue image data and $F = \sum_{m=1}^{M} a_m f_m.$6 with solely RNA-seq data. The same study reported $F = \sum_{m=1}^{M} a_m f_m.$7 for raw concatenation, $F = \sum_{m=1}^{M} a_m f_m.$8 for concatenation of learned embeddings, and $F = \sum_{m=1}^{M} a_m f_m.$9 for mean-vector fusion, reinforcing the point that modality-wise focus is not reducible to static aggregation (Deng et al., 2023).

In detection, DEYOLO’s ablation on $S_A$0FD moved from a visible-only YOLOv8n baseline of mAP$S_A$1, mAP$S_A$2 to a full DEYOLO-n score of $S_A$3, with intermediate gains from bi-direction focus, DECA, and DEPA. The same report compared a best prior image-fusion-plus-YOLO result of approximately $S_A$4 with DEYOLO-l at $S_A$5, supporting the detection-centered feature-fusion interpretation of CMFF (Chen et al., 2024).

FocusMamba provides a complementary efficiency result. Its base VSS backbone with simple fusion achieved $S_A$6 mAP$S_A$7/mAP at $S_A$8G FLOPs, EGMS alone improved this to $S_A$9 at $S_B$0G, CAE raised it to $S_B$1 at $S_B$2G, and the full CMFF with FI-Mamba reached $S_B$3 at $S_B$4G. The same paper reported that ConcatMamba and CrossMamba remained around $S_B$5–$S_B$6 mAP$S_B$7 for similar FLOPs, indicating that sparse, complementarity-aware fusion, not merely the use of Mamba, accounts for the gain (Yang et al., 4 Sep 2025).

Fusion-Mamba reported that it outperforms state-of-the-art methods on $S_B$8AP with $S_B$9 on $S_F$0 and $S_F$1 on FLIR-Aligned, and its internal ablations showed that removing SSCS or DSSF degrades performance, with full FLIR-Aligned results of $S_F$2 mAP$S_F$3 and $S_F$4 mAP versus $S_F$5 when both are removed (Dong et al., 2024). SFEDet, by contrast, achieved competitive accuracy with substantially fewer parameters and lower cost; on M3FD it reported $S_F$6M parameters, $S_F$7G FLOPs, AP50 $S_F$8, and mAP $S_F$9, while dense FPN-level DEGF fusion used $S_A$0G FLOPs for AP50 $S_A$1 and mAP $S_A$2 (Tian et al., 29 Jun 2026).

For unified latent-space fusion, $S_A$3-Fusion reported strong gains under corruption. Under EKubric under-exposure, 2D EPE improved from $S_A$4 to $S_A$5, and ACC$S_A$6 improved from $S_A$7 to $S_A$8. Under DSEC-PC LiDAR drifting, 3D EPE improved from $S_A$9 to $S_B$0, while ACC$S_B$1 improved from $S_B$2 to $S_B$3. Its ablations further showed that removing Event Edge Space worsened EPE$S_B$4 from $S_B$5 to $S_B$6 and EPE$S_B$7 from $S_B$8 to $S_B$9 (Guo et al., 17 Mar 2026).

6. Conceptual boundaries, limitations, and open directions

Several misconceptions recur in the interpretation of CMFF. First, CMFF is not synonymous with Transformer cross-attention. The NSCLC survival model explicitly described its fusion as cross-modality attention while also noting that it is not Transformer query–key–value attention (Deng et al., 2023). Second, CMFF is not equivalent to image-space fusion. DEYOLO and SFEDet both argue for feature-space or proposal-space fusion that is explicitly task-centered, rather than visually plausible fused images (Chen et al., 2024, Tian et al., 29 Jun 2026). Third, CMFF need not be dense. Token-sparse and proposal-sparse designs show that focused fusion can be more accurate and more efficient when most regions are background (Yang et al., 4 Sep 2025, Tian et al., 29 Jun 2026).

The literature also exposes unresolved design questions. An early multimodal attention paper already emphasized that the stage at which attention should be used was under scrutiny (Panchal, 2020). Later systems answered this differently: encoder-side and decoder-side integration in vision–LLMs, mid–late feature fusion in RGB–IR detection, post-encoder alignment in flow estimation, and coarse-to-fine interaction in registration (Liu et al., 14 Apr 2025, Guo et al., 17 Mar 2026, Li et al., 2 Jun 2026). This suggests that the placement of focus and fusion remains task-dependent rather than settled.

Limitations are equally heterogeneous. Some methods do not evaluate missing-modality settings even though their formulations could, in principle, support masking or uncertainty-aware weighting (Deng et al., 2023). State-space and Mamba-based systems can inherit implicit-correlation modeling or absence of explicit cross-modal regularization, which may leave complementarity underconstrained (Xie et al., 2024). Large integrated models require trade-offs in latent-token count, window size, or recursive depth, and can remain question-specific or computationally delicate despite improved alignment (Liu et al., 14 Apr 2025).

Current trajectories point toward three broad directions. One is stronger representation unification, as in Event Edge Space, where a physically meaningful modality anchors a homogeneous latent space (Guo et al., 17 Mar 2026). A second is more selective compute allocation, through sparsification, sparse RoIs, or adaptive token budgets (Yang et al., 4 Sep 2025, Tian et al., 29 Jun 2026). A third is hierarchical focus spanning within-modality structure, cross-modality alignment, and cross-task coupling, as seen in registration, survival prediction, and multimodal language–vision models (Li et al., 2 Jun 2026, Deng et al., 2023, Liu et al., 14 Apr 2025). A plausible implication is that future CMFF systems will increasingly combine these axes—shared latent spaces, sparse computation, and hierarchical attention—rather than treating fusion as a single isolated layer.