CMIM for Modality Identity in Medical Synthesis

• CMIM is a semantic regularizer that enforces causal modality identity in Med-K2N by aligning image features with target modality descriptions.
• It employs contrastive and metric learning losses to reduce identity drift in multi-output synthesis across modalities like MRI and CT.
• Empirical evaluations demonstrate that incorporating CMIM improves PSNR and SSIM, enhancing the diagnostic fidelity of synthesized images.

The Causal Modality Identity Module (CMIM) is a component of Med-K2N, a framework for flexible $K$-to-$N$ modality translation in medical image synthesis, where available input modalities are used to reconstruct multiple missing target modalities (Yuan et al., 3 Oct 2025). Within that system, CMIM is the mechanism specifically intended to maintain modality identity consistency in multi-output generation by establishing causal constraints between each generated image and a target modality description in a shared vision–language embedding space. Its purpose is to reduce identity drift, such as contamination of a synthetic $T1$ image by $T2$-like features, while complementing Med-K2N’s other modules for modality–task weighting and fusion quality control (Yuan et al., 3 Oct 2025).

1. Position within K-to-N modality translation

In Med-K2N, the synthesis problem is formulated with available input modalities $X = \{x_m\}_{m \in M_{in}}$ and target modalities $Y = \{y_n\}_{n \in N_{out}}$, with the overall mapping written as

$$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$

Inputs are encoded to a shared feature space and progressively fused, and each $Y_j$ is generated by a task-specific head (Yuan et al., 3 Oct 2025).

The paper identifies three challenges in this setting: modeling modality–task heterogeneity, controlling fusion quality, and maintaining modality identity consistency. The first two are addressed by PreWeightNet, ThresholdNet, and EffiWeightNet, together with quality-driven acceptance gates and TaskHeadNet’s multi-candidate selection. CMIM addresses the third challenge: ensuring that each synthesized output $Y_j$ matches the semantic identity of its intended modality, rather than merely matching pixel-level or feature-level similarity (Yuan et al., 3 Oct 2025).

The rationale is clinical as well as algorithmic. Conventional fusion and alignment methods may optimize reconstruction quality without explicitly enforcing the modality’s semantic identity. The paper states that such identity drift undermines clinical safety, because downstream diagnosis depends on modality-specific contrast mechanisms. In that context, CMIM functions as a semantic regularizer on top of the synthesis pipeline, constraining the generator so that multi-output reconstruction does not blur the distinctions among modalities such as MRI $T1$-weighted, MRI $N$0-weighted, FLAIR, CT, or PET (Yuan et al., 3 Oct 2025).

2. Causal framing and semantic identity constraint

CMIM’s core intuition is to tie each generated image $N$1 to a target modality description $N$2 in a shared semantic space learned by a medical-domain vision–LLM. The paper describes the intended causal chain as “modality type $N$3 image features $N$4 semantic expression,” and proposes that alignment between image and text embeddings encourages the generator to produce visual features causally consistent with the target modality type (Yuan et al., 3 Oct 2025).

The structural causal perspective is described in words rather than through formal causal identification. The relevant variables include source modalities $N$5, fusion weights $N$6, generated image $N$7, target modality description text $N$8, and confounders or noise $N$9, such as motion, scanner variability, or pathology appearance that might mimic other modalities. The assumed edges are: $T1$0 encoding/fusion with $T1$1; latent modality type $T1$2 visual features of $T1$3; $T1$4; visual features of $T1$5 semantic embedding $T1$6; and text $T1$7 textual embedding $T1$8 (Yuan et al., 3 Oct 2025).

The paper explicitly distinguishes this use of “causal constraints” from purely correlational alignment. In practice, the causal intent is operationalized by loss terms that pull the generated image embedding toward the correct modality text and real images of the same modality, while pushing it away from embeddings associated with other modality texts and images. The paper also states that it does not formalize do-operator or backdoor adjustment. A plausible implication is that CMIM should be understood as causal framing implemented through contrastive and metric-learning constraints, rather than as a fully identified structural causal model (Yuan et al., 3 Oct 2025).

This distinction matters because the acronym “CMIM” is already associated with “Cross-Modal Information Maximization,” a 2020 method that maximizes mutual information between modality-specific and multimodal representations (Sylvain et al., 2020). That earlier CMIM does not introduce causal assumptions and does not include a Causal Modality Identity Module. In Med-K2N, the acronym refers to a different object entirely: a modality identity regularizer grounded in vision–language alignment rather than mutual-information estimation (Yuan et al., 3 Oct 2025, Sylvain et al., 2020).

3. Architecture and mechanism

CMIM is implemented as a vision–text dual-encoder module, described as akin to CLIP-like encoders, that maps generated images and target modality descriptions to a common embedding space. The image encoder $T1$9 produces an embedding $T2$0 from $T2$1, and the text encoder $T2$2 produces $T2$3 from $T2$4 (Yuan et al., 3 Oct 2025).

The paper states that a medical domain pre-trained vision–LLM is used, but it does not specify the exact encoder architectures, such as ViT, ResNet, or BERT, and it does not specify whether the encoders are frozen or fine-tuned during CMIM training. It also states that attention or cross-modal fusion is not used inside CMIM; the module provides an external semantic constraint through its loss terms rather than an internal cross-modal fusion block (Yuan et al., 3 Oct 2025).

Target modality descriptions are referred to as “target modality descriptive texts” $T2$5. Concrete prompt templates are not listed, and the paper does not state whether prompts are hand-crafted or learned. It does, however, indicate that in practice these can be standardized prompts for modalities such as CT, MRI $T2$6/FLAIR, and PET, optionally augmented with anatomical or task context (Yuan et al., 3 Oct 2025).

Within Med-K2N, CMIM is attached after TaskHeadNet produces candidate outputs. For each target modality $T2$7, TaskHeadNet generates the final $T2$8 after multi-candidate selection by a quality evaluator. CMIM then encodes $T2$9 and its corresponding text description $X = \{x_m\}_{m \in M_{in}}$0 to compute modality identity losses. Gradients from those losses flow back to the generator components, influencing MultiScaleNet and TaskHeadNet, and indirectly affecting fusion weights predicted by PreWeightNet, ThresholdNet, and EffiWeightNet through the quality loop (Yuan et al., 3 Oct 2025).

CMIM therefore complements rather than replaces the fusion subsystem. PreWeightNet, ThresholdNet, and EffiWeightNet determine what to fuse, whether to accept it, and how strongly to fuse it, whereas CMIM constrains what the target modality must be. This separation of roles is central to the system design: fusion modules manage contribution and quality, while CMIM penalizes visual–semantic deviations from the intended modality identity (Yuan et al., 3 Oct 2025).

4. Mathematical formulation and optimization

CMIM uses image and text embeddings

$X = \{x_m\}_{m \in M_{in}}$1

together with cosine similarity

$X = \{x_m\}_{m \in M_{in}}$2

Its primary alignment term is an InfoNCE/CLIP-style contrastive loss:

$X = \{x_m\}_{m \in M_{in}}$3

where $X = \{x_m\}_{m \in M_{in}}$4, $X = \{x_m\}_{m \in M_{in}}$5, and $X = \{x_m\}_{m \in M_{in}}$6 is the temperature (Yuan et al., 3 Oct 2025).

A second term strengthens discriminative separation through metric learning:

$X = \{x_m\}_{m \in M_{in}}$7

where $X = \{x_m\}_{m \in M_{in}}$8 is a distance in the embedding space, $X = \{x_m\}_{m \in M_{in}}$9 is an embedding of a real target image of modality $Y = \{y_n\}_{n \in N_{out}}$0, and $Y = \{y_n\}_{n \in N_{out}}$1 is an embedding from an incorrect modality $Y = \{y_n\}_{n \in N_{out}}$2 (Yuan et al., 3 Oct 2025).

The full Med-K2N objective is written as

$Y = \{y_n\}_{n \in N_{out}}$3

with $Y = \{y_n\}_{n \in N_{out}}$4 corresponding to the CMIM alignment loss, written in the paper as $Y = \{y_n\}_{n \in N_{out}}$5 (Yuan et al., 3 Oct 2025). The paper does not specify the values of $Y = \{y_n\}_{n \in N_{out}}$6, $Y = \{y_n\}_{n \in N_{out}}$7, or the $Y = \{y_n\}_{n \in N_{out}}$8 coefficients. It notes only that common practice is to tune the CMIM weights so that identity regularization is sufficiently strong without overwhelming reconstruction fidelity. That statement is not a reported hyperparameter choice but practical guidance recorded in the detailed explanation (Yuan et al., 3 Oct 2025).

The paper also notes that consistency across multiple outputs is handled naturally by the $Y = \{y_n\}_{n \in N_{out}}$9-way softmax normalization in $$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$0, because incorrect modality texts $$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$1 act as shared negatives. This discourages identity confusion among concurrent output heads in the multi-output setting (Yuan et al., 3 Oct 2025).

5. Integration with sequential fusion and training regime

Med-K2N is inspired by SAM2’s sequential frame paradigm and a clinician workflow of progressively adding and selectively integrating multimodal information. Inputs are treated as sequential frames with a key-frame baseline and auxiliary progressive enhancement. CMIM operates on the final generated frames for each target modality, rather than inside the sequential fusion mechanism itself (Yuan et al., 3 Oct 2025).

The upstream weighting and gating modules are mathematically specified. PreWeightNet learns global contribution weights $$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$2 for each source–target pair using task-aware encodings and a task-specific memory bank:

$$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$3

$$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$4

ThresholdNet learns adaptive acceptance thresholds

$$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$5

$$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$6

with $$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$7 and $$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$8. EffiWeightNet then computes effective spatial fusion weights:

$$F: \{X_1, X_2, \ldots, X_K\} \to \{Y_1, Y_2, \ldots, Y_N\}.$$9

$Y_j$0

TaskHeadNet fuses baseline and weighted auxiliary features, generates multiple candidates per target, and selects the best with a quality evaluator (Yuan et al., 3 Oct 2025).

Training uses PyTorch on NVIDIA A100 GPUs. The SAM2 image encoder is LoRA+ fine-tuned to medical features, other modules are Kaiming-initialized, slices are resampled to $Y_j$1, and augmentations include horizontal flip with $Y_j$2, gentle color jitter, Gaussian blur with $Y_j$3, random resized cropping with scale $Y_j$4–$Y_j$5, and identity normalization. The schedule is $Y_j$6 epochs with cosine learning rate starting at $Y_j$7, batch size $Y_j$8 per GPU, and gradient accumulation of $Y_j$9 (Yuan et al., 3 Oct 2025).

The curriculum is four-stage, easy-to-expert, with ratios $Y_j$0, and loss masks progressively enable perceptual, causal, and quality-aware terms as task difficulty increases. The paper states that this stabilizes convergence. A plausible implication is that CMIM is not merely an auxiliary loss but part of a staged optimization strategy designed to prevent early training instability in multi-output synthesis (Yuan et al., 3 Oct 2025).

6. Empirical evidence, novelty, and limitations

The empirical evaluation uses two datasets: a Combined Brain Tumor Dataset consisting of BraTS2019, BraTS-MEN, and BraTS-MET with $Y_j$1 patients and modalities $Y_j$2, $Y_j$3, $Y_j$4, and $Y_j$5; and ISLES 2022 with $Y_j$6 multi-center MRI cases containing DWI, ADC, and FLAIR. The reported synthesis metrics are PSNR and SSIM (Yuan et al., 3 Oct 2025).

Across many $Y_j$7 settings, Med-K2N achieves the best PSNR and SSIM compared with MM-Synthesis, pGAN, and MM-Transformer, with entries marked by “*” indicating baselines that are significantly worse at $Y_j$8 under the Wilcoxon signed-rank test. For the ablation on $Y_j$9 synthesis from $T1$0 in the merged brain tumor dataset, the sequence of results is: baseline fusion at $T1$1 PSNR and $T1$2 SSIM; $T1$3 at $T1$4 and $T1$5; $T1$6 at $T1$7 and $T1$8; $T1$9 at $N$00 and $N$01; $N$02 at $N$03 and $N$04; and $N$05 at $N$06 and $N$07. The CMIM increment is therefore $N$08 dB PSNR and $N$09 SSIM within that ablation trajectory (Yuan et al., 3 Oct 2025).

Qualitatively, the paper states that CMIM prevents modality identity confusion, including $N$10-like features in synthetic $N$11 and MRI-like contrast in synthetic CT, and that visual examples show sharper lesion boundaries and appropriate contrast characteristics when CMIM is active and more inputs are fused (Yuan et al., 3 Oct 2025). The paper does not report text–image retrieval or modality classification metrics, so the evidence for identity consistency is indirect, through improved synthesis fidelity and ablation gains (Yuan et al., 3 Oct 2025).

In comparative positioning, the paper argues that cycle-consistency and GAN-based constraints can improve mapping consistency but often lack explicit modality identity semantics in multi-output settings; classifier guidance or domain-specific feature losses may impose only weak or indirect identity constraints; and CLIP guidance without causal framing may treat identity as correlation. CMIM’s stated novelty is to bring modality-aware semantic supervision into multi-output medical synthesis via a medical VLM and explicit $N$12-way contrastive plus metric losses, making identity consistency an explicit objective alongside reconstruction fidelity and SSIM (Yuan et al., 3 Oct 2025).

The limitations are also explicit. CMIM assumes that target descriptions accurately reflect modality and clinical context; poor or ambiguous prompts can weaken identity constraints. It reduces identity drift but does not eliminate the effect of noisy inputs or artifacts, so robust fusion and gating remain essential. The evaluation is limited to brain imaging, and broader validation is needed for other anatomies and modalities, including PET-CT beyond the brain. Dual encoders add compute, memory, and latency overhead, especially as the number of target modalities $N$13 grows. Finally, the paper does not specify $N$14, $N$15, or the loss weights, so deployment requires tuning and validation monitoring (Yuan et al., 3 Oct 2025).

A recurrent misconception is that CMIM refers unambiguously to a single method family. In the arXiv literature, “CMIM” also denotes “Cross-Modal Information Maximization,” which is a mutual-information-based framework for resilient single-modality inference under modality dropout and explicitly does not involve causal modeling or a Causal Modality Identity Module (Sylvain et al., 2020). In the Med-K2N context, CMIM refers specifically to the causal identity-consistency module described above (Yuan et al., 3 Oct 2025).