Dual-Modal CT–CL Fusion Strategy

• Dual-modal CT–CL fusion strategy integrates high-resolution CT with clinical data to leverage complementary global statistics and local structural details.
• Multiple frameworks including deep reliability-weighted fusion, coupled dictionary learning, and evidential reasoning are employed to balance feature retention and uncertainty assessment.
• Practical implementations demonstrate improved segmentation, classification, and robustness against input degradation, supporting enhanced clinical decision-making.

A dual-modal CT–CL fusion strategy integrates computed tomography (CT) with a complementary clinical (CL) modality—such as clinical images, clinical data, contrast-enhanced imaging, or other relevant channels—to yield a representation or prediction that combines the unique information offered by each modality. These strategies address the trade-offs inherent in single-modality approaches, including global statistical similarity, local detail preservation, robustness to degraded input channels, explicable decision logic, and uncertainty quantification. Multiple methodological paradigms exist, including deep learning–based spatial-frequency fusion, dictionary learning, evidential reasoning, and hybrid attention mechanisms.

1. Architectural Principles and Motivation

Dual-modal CT–CL fusion approaches are motivated by the complementary properties of CT (typically high-resolution anatomical morphometry or radiodensity) and the CL channel (providing functional, semantic, or domain-specific context). The main objective is to maximize global statistical similarity (e.g., correlation, mutual information) with the input modalities while enhancing local structural/semantic fidelity and permitting explicit reasoning about uncertainty and interpretability.

Modern architectures leverage modular design including modality-specific encoders, reliability or attention weighting, expert fusion modules (spatial and frequency domains), and dedicated loss function design to resolve the central trade-off between retaining shared content and maximizing detail (Islam, 13 Jan 2026).

2. Key Methodological Frameworks

Several key algorithmic designs dominate the literature:

2.1 Reliability-Weighted Dual-Expert Fusion (e.g., W-DUALMINE)

This paradigm employs:

• Siamese multi-scale encoders to extract feature pyramids {f₁ˢ, f₂ˢ} from each modality.
• Reliability maps, computed via softplus-activated convolutions, estimate location- and scale-dependent weights for each modality.
• Dual-expert fusion at each scale, with a spatial expert (dilated convolutions for context) and a wavelet-domain frequency expert (detail band fusion by max-magnitude and low-band by reliability).
• Soft gradient-based arbitration fuses expert outputs using gradient magnitude–weighted softmax.
• Residual-to-average fusion: the fused output is a residual map (learned local differences) plus the pixelwise average of CT and CL, which maximizes the fused image’s linear correlation with both sources.
• Loss design explicitly balances correlation coefficient (CC), mutual information (MI via InfoNCE), and structural fidelity (gradient-max loss), with a final objective of

$$\mathcal{L} = \lambda_{CC}\,\mathcal{L}_{CC} + \lambda_{MI}\,\mathcal{L}_{MI} + \lambda_s\,\mathcal{L}_s.$$

This approach demonstrates consistent improvement in both CC and MI metrics across CT–CL datasets (Islam, 13 Jan 2026).

2.2 Dictionary-Based Coupled Feature Fusion

The coupled dictionary learning (CDL) strategy (Veshki et al., 2021) decomposes patches of CT and CL images into coupled and independent components:

• Coupled (shared) parts modeled as sparse codes on paired dictionaries, enforcing support congruence to capture shared structure.
• Independent parts capture modality-specific edges, noise, or artifacts, penalized by squared Pearson correlation to maximize statistical independence.
• The final fusion rule selects dominant coefficients (max-absolute-value) to reconstruct a composite image that preserves salient aspects of both modalities.
• Alternating minimization with sparse coding (modified SOMP) and independent-part EM update ensures convergence.

2.3 Evidential Reasoning Fusion (AER/ER)

Analytic evidential reasoning fusion (Zhou et al., 2017, Huang et al., 2023) treats each modality’s classifier output as “evidence,” assigning both a weight (modality importance) and a reliability (agreement/consistency):

• Outputs are formalized as belief mass assignments over potential hypotheses (e.g., disease presence).
• The analytic ER rule combines masses using closed-form orthogonal-sum equations, balancing confidence and evidential conflict:

$p_{fused} = \frac{m(h_1)}{m(h_1) + m(h_2)}$

Reliability terms $r_i$ directly down-weight outlier or discordant modalities.

• This approach is robust to missing modalities, preserves sensitivity/specificity trade-offs, and scales to multi-modality settings.

2.4 Contrastive Learning and Intermediate Fusion

Multistage pre-training with contrastive objectives has proven highly effective (Jung et al., 22 Jan 2025, Ruffini et al., 15 Jan 2026):

• SimCLR-based pre-training on CT slices builds strong CT embeddings.
• Cross-modal contrastive alignment matches CT and CL embeddings in shared latent space, using symmetric NT-Xent or InfoNCE loss.
• Intermediate fusion concatenates downstream CT and CL vectors (optionally in NAIM or ABMIL frameworks) followed by MLP or ODST survival heads.
• Adaptive feature importance: fusion heads learn to dynamically weigh stronger modalities and “gate out” less informative or noisy sources.

3. Implementation Paradigms and Training Pipelines

The dual-modal CT–CL workflow is context-responsive and involves several pipeline stages:

Preprocessing and Normalization

• CT volumes: resampled to uniform voxel size, normalized to [0,1] via Hounsfield unit windowing, optionally histogram-matched.
• CL images/clinical data: modality-specific normalization, e.g., luminance channels, z-scoring, or one-hot/ordinal encodings.

Feature Extraction

• Modality-specific encoders: 3D U-Nets, ResNet50-based 2D encoders (for slices), vision transformers, or clinical MLP/tokenization branches.
• Reliability estimation (for deep fusion) or explicit reliability calculation (for evidential methods).

Fusion Operations

• Dual-expert (spatial/frequency domain), early, intermediate, or late-stage fusion via concatenation, attention, or analytic rules.
• Arbitration modules (e.g., soft gradient-based attention or dynamic attention modules) to resolve conflicts and select salient features adaptively.

Training and Optimization

• Multi-objective losses: combine global statistics (correlation, MI), local structure (gradient, Dice), and cross-modal contrastive alignment.
• Regularization by smoothness, inverse-consistency, or volume-preservation when registration is involved.
• Hyperparameter tuning (e.g., $\lambda$ weights, batch size, learning rates) tailored to modality characteristics.

Performance Outcomes

Consistent quantitative improvements are reported:

• For fusion quality: increases in Dice (e.g., by +0.04–0.14 for segmentation (Tschuchnig et al., 2024), +4.1 pp over naive early-fusion in evidential models (Huang et al., 2023)), CC/MI, and AUROC (e.g., +0.06–0.08 in contrastive models (Jung et al., 22 Jan 2025)).
• For decision support: improved AUC for disease prediction (e.g., ER-fusion AUC = 0.87 vs. 0.80 for best single-modality (Zhou et al., 2017)).
• For robustness: adaptive down-weighting of less informative modalities and resilience to missing channels (Ruffini et al., 15 Jan 2026), as well as clear evidence of improved performance under degraded input conditions (e.g., heavy CBCT artifacts (Tschuchnig et al., 2024)).

4. Comparative Analysis Across Fusion Strategies

A spectrum of fusion strategies is employed depending on the scientific and clinical objective:

Approach	Modality Encoding	Fusion Stage	Reliability Handling
Dual-expert deep fusion (Islam, 13 Jan 2026)	Multi-scale encoders	Feature: multi-level, residual-based	Learned dense reliability maps
Coupled dictionary (Veshki et al., 2021)	Patch-wise sparse	Feature, patch-wise	Pearson-correlation penalty
Evidential reasoning (Zhou et al., 2017, Huang et al., 2023)	Feature extraction + classifier	Score/decision layer	Analytic reliabilities (agreement)
Contrastive/intermediate (Jung et al., 22 Jan 2025, Ruffini et al., 15 Jan 2026)	Deep FM/MLP embeddings	Intermediate fusion (concat/attention)	Dynamic gating via ODST/attention
Early-fusion 3D U-Net (Tschuchnig et al., 2024)	Channel stacking	Input layer	No explicit reliability, U-Net adapts

Hybrid approaches are increasingly integrating evidential reliability, deep feature fusion, and explicit contrastive alignment to simultaneously capture global similarity, preserve sharp details, and provide robust, calibrated uncertainty estimation.

5. Practical Adaptation and Modality-Specific Considerations

Dual-modal fusion pipelines require discipline-specific adaptation:

• CT edge structure: sharp, high-frequency, favors detail-preserving wavelet bases (e.g., Daubechies-4 in (Islam, 13 Jan 2026)).
• CL softness and noise: luminance/contrast matching, histogram leveling, and basis selection for low-frequency stability.
• Registration: critical in settings with significant resolution mismatch (e.g., $\mu$CT vs. clinical CT (Roth et al., 2017)), requiring multi-scale free-form deformation models and similarity measures appropriate to modality statistics (NMI, cross-correlation).

Hyperparameters such as fusion weights, reliability stabilizers, augmentation parameters, and learning rates are tuned specifically, especially in the presence of severe artifacts or imperfect registration (e.g., synthetic affine+elastic misalignment (Tschuchnig et al., 2024)).

6. Evaluation Metrics and Experimental Findings

A consistent set of metrics is used for performance benchmarking:

• Segmentation: Dice coefficient, Brier score, negative log-likelihood (NLL), expected calibration error (ECE).
• Classification/regression: AUROC, C-index (survival), accuracy.
• Uncertainty quantification: mass on ignorance (DST), conflict, aleatory and epistemic decomposition.

Empirical findings confirm that dual-modal CT–CL fusion:

• Substantially improves performance over single-modality and naive concatenation—e.g., +0.14 Dice on severely degraded CBCT tumor segmentation, up to +0.06–0.08 AUROC in cross-modal contrastive learning (Jung et al., 22 Jan 2025, Tschuchnig et al., 2024).
• Achieves robustness to missing modalities, noise, and misalignment by adaptive reliability weighting or deep evidential aggregation (Huang et al., 2023, Ruffini et al., 15 Jan 2026).
• Provides interpretability via explicit reliability and feature importance traces, supporting clinical translation (Zhou et al., 2017).

7. Future Directions and Open Issues

Promising directions for CT–CL fusion include:

• Extending evidential and reliability-based frameworks to more than two modalities with generalized weighting schemes.
• Incorporation of spatial transformer networks or attention-based co-registration to refine alignment in real-world clinical datasets (Tschuchnig et al., 2024).
• Large-scale contrastive pre-training and foundation models for universal feature spaces spanning anatomical, functional, and semantic domains (Jung et al., 22 Jan 2025, Ruffini et al., 15 Jan 2026).
• Development of uncertainty-aware clinical decision systems where calibrated belief and ignorance mass can be explicitly leveraged for risk stratification.

Ongoing challenges include harmonizing modalities with disparate spatial/temporal resolutions and data missingness, automating reliability estimation, and ensuring computational tractability in high-dimensional fusion setups. The dual-modal CT–CL paradigm provides a robust and extensible framework for integrated, explainable, and high-fidelity clinical image and data analysis across diverse application domains.