Med-K2N: Flexible K-to-N Image Synthesis

• Med-K2N is a flexible K-to-N framework that synthesizes missing medical imaging modalities from an arbitrary set of inputs to address incomplete clinical exams.
• It employs a progressive fusion strategy with sequential integration using modules like PreWeightNet, ThresholdNet, and EffiWeightNet for adaptive, modality-specific weighting.
• The architecture enforces modality identity with vision-language constraints and demonstrates superior performance on brain imaging datasets over traditional fixed-pair methods.

Searching arXiv for the Med-K2N paper and closely related works named in the provided data. Med-K2N denotes a flexible medical image synthesis framework that takes an arbitrary set of available imaging modalities as inputs and synthesizes an arbitrary set of missing modalities in a single, unified model (Yuan et al., 3 Oct 2025). It formulates cross-modal synthesis as a general mapping

$$\mathcal{F}: \{X_1, X_2, \ldots, X_K\} \rightarrow \{Y_1, Y_2, \ldots, Y_N\},$$

with variable $K$ and variable $N$, thereby extending beyond conventional $1\!\to\!1$, $K\!\to\!1$, and $1\!\to\!K$ settings. The framework is designed for clinical environments in which exams are incomplete, acquisition time and cost are constrained, and different modalities contribute complementary but heterogeneous diagnostic information. The same string has also appeared in a separate condensed-matter context associated with a K$_2$N monolayer, a two-dimensional $d^0$ ferromagnetic Weyl nodal-line semimetal (Jin et al., 2020); in the medical-imaging literature, however, Med-K2N refers to the K-to-N modality translation framework (Yuan et al., 3 Oct 2025).

1. Problem setting and motivating constraints

Med-K2N is motivated by the observation that clinical image acquisition often fails to provide a complete modality set because of limited scanner time and patient fatigue, cost and equipment availability, and contraindications such as contrast agents or radiation exposure (Yuan et al., 3 Oct 2025). Within this setting, synthesis of missing modalities is framed not as a fixed-pair translation problem but as a general modality-completion problem in which both the observed sources and the desired targets may vary case by case.

The framework is explicitly positioned against three limitations identified in prior work. First, single- or fixed-pair synthesis cannot adapt to heterogeneous clinical scenarios where the available modalities vary. Second, uniform fusion assumes that all source modalities contribute similarly to all target tasks, whereas the paper states that different input modalities contribute differently to each target modality. Third, many sequential approaches inspired by SAM process data in a sequence-like fashion but produce only one output modality rather than multiple parallel outputs.

These limitations are condensed into three central challenges. The first is modeling heterogeneous modality-task contributions: for a given target, some sources are substantially more informative than others. The second is fusion quality control: noisy or unhelpful modalities can degrade the synthesized result if there is no mechanism for real-time evaluation of their utility. The third is modality identity consistency in multi-output generation: without explicit constraints, a synthesized image for one modality can exhibit the contrast characteristics of another modality. Med-K2N is organized around these three problems rather than treating multimodal fusion as a fixed aggregation operator.

2. Architectural formulation and sequential processing

The architecture is built around two linked inspirations: SAM2’s sequential frame paradigm and clinicians’ progressive workflow of incrementally adding and selectively integrating multimodal information (Yuan et al., 3 Oct 2025). Multi-modal medical images of the same anatomy are treated as a temporal sequence. One modality is designated as the key frame, providing a baseline representation, while the remaining modalities are treated as auxiliary frames that are integrated one by one through a step-by-step enhancement process.

The first stage is a LoRA-fine-tuned SAM2 encoder. Each modality slice or volume is processed as a separate frame, and the encoder produces multi-scale feature maps $F_i \in \mathbb{R}^{H \times W \times D}$ for each modality $X_i$. These features are then passed to MultiScaleNet, which builds a feature pyramid and uses bidirectional Mamba modules with a Fermat-spiral scanning strategy for efficient, direction-unbiased context modeling. MultiScaleNet produces key features $K$0 and auxiliary features $K$1.

The central fusion stage is a progressive cross-modal fusion network composed of PreWeightNet, ThresholdNet, and EffiWeightNet. These modules operate sequentially for each auxiliary modality and for each target modality, implementing what the paper describes as “primary frame baseline + auxiliary modality step-by-step enhancement.” Downstream of fusion, TaskHeadNet receives the fused representation and generates each target modality. It contains multiple generation heads per task and an internal quality feedback mechanism that selects the best candidate and feeds quality information back to the fusion modules. To enforce modality identity consistency, the architecture adds the Causal Modality Identity Module, or CMIM, a vision-language module that constrains generated images to align with their intended modality descriptions.

At the pipeline level, the model first encodes the available inputs $K$2, forms a baseline prediction from the key-frame feature $K$3, then progressively evaluates each auxiliary modality $K$4 relative to each target $K$5. For each modality-task pair, PreWeightNet computes a global importance weight, ThresholdNet computes an adaptive threshold, and EffiWeightNet computes an effective spatial weight map. TaskHeadNet then fuses the weighted auxiliary features into a shared representation $K$6, generates multiple candidates $K$7, scores them through a quality assessment module, and outputs the selected $K$8. CMIM supplies additional modality-consistency constraints through vision-language supervision.

3. Progressive fusion and the three collaborative modules

The fusion mechanism is defined at the granularity of source-modality/target-task pairs $K$9, so that each available modality can be evaluated differently for each synthesis objective (Yuan et al., 3 Oct 2025). This pairwise treatment is the technical basis for the framework’s claim of flexible $N$0 synthesis.

PreWeightNet performs global contribution assessment. Its TaskAware Feature Encoder integrates baseline features $N$1, auxiliary features $N$2, and previous outputs-related information into task-aware features $N$3. For each task $N$4, it also maintains a learnable memory matrix

$N$5

which stores task-specific successful fusion patterns. A task query $N$6 is formed as

$N$7

and relevant memory is retrieved by attention: $N$8 The final global importance score is then

$N$9

where $1\!\to\!1$0 quantifies the overall importance of modality $1\!\to\!1$1 for generating $1\!\to\!1$2.

ThresholdNet performs adaptive filtering. It fuses the global importance $1\!\to\!1$3, retrieved task memory $1\!\to\!1$4, modality compatibility $1\!\to\!1$5, and performance history $1\!\to\!1$6. Compatibility is encoded as

$1\!\to\!1$7

and the gating representation is

$1\!\to\!1$8

The adaptive threshold is predicted as

$1\!\to\!1$9

with $K\!\to\!1$0 and $K\!\to\!1$1. In the paper’s interpretation, higher $K\!\to\!1$2 imposes stricter acceptance criteria and stronger filtering, whereas lower $K\!\to\!1$3 permits more inclusive fusion.

EffiWeightNet computes the final effective fusion weights. Its Fusion Weight Calculator integrates global weights, thresholds, task memory, gating features, task embedding, and modality embedding: $K\!\to\!1$4 The effective weights are then given by

$K\!\to\!1$5

where $K\!\to\!1$6 is a small constant such as $K\!\to\!1$7. This avoids extreme binary decisions and stabilizes training while preserving task-specific and potentially spatially varying weighting.

Taken together, the three modules implement a cascade: PreWeightNet asks whether a modality is globally useful for a target, ThresholdNet determines the acceptance criterion under current quality and historical conditions, and EffiWeightNet produces the final weight map used in fusion. The paper presents this sequence as an operational analogue of a clinician who adds one imaging sequence at a time and selectively incorporates only the beneficial information.

4. CMIM, modality identity, and training objectives

CMIM addresses the problem that visually plausible synthesized images may nevertheless fail to match their declared modality identity (Yuan et al., 3 Oct 2025). The paper frames the issue causally through the chain

$K\!\to\!1$8

Under this view, modality type determines acquisition-specific image statistics, and those image features should align with modality-specific semantic descriptions.

Implementation-wise, CMIM uses a dual-encoder structure similar to CLIP: $K\!\to\!1$9 where $1\!\to\!K$0 is the generated image for target modality $1\!\to\!K$1 and $1\!\to\!K$2 is the textual description of that modality. Alignment is enforced first through a cross-modal contrastive loss,

$1\!\to\!K$3

which encourages each generated image to match its own modality description while separating it from descriptions of other modalities.

A second constraint is a metric-learning loss. Using the generated embedding $1\!\to\!K$4, a reference embedding $1\!\to\!K$5 from a real image of the same modality, and a negative embedding $1\!\to\!K$6 from a different modality, CMIM minimizes

$1\!\to\!K$7

This makes generated images cluster with real images of the same modality and remain separated from images of other modalities, thereby reducing identity blending across outputs.

The overall training objective combines four losses: $1\!\to\!K$8 Here $1\!\to\!K$9 is the pixel-level reconstruction loss, $_2$0 preserves structural similarity, $_2$1 corresponds to the vision-language contrastive term, and $_2$2 is the metric-learning term. The paper states that no explicit adversarial loss is used.

Optimization is organized through a 100-epoch curriculum split into four stages with ratios $_2$3. The easy stage includes only cross-modal $_2$4 mappings excluding identity targets; the medium stage introduces multimodal fusion $_2$5; the hard stage expands to $_2$6; and the expert stage trains on full $_2$7 patterns with disjoint input and target sets. The curriculum controller is seeded by epoch, batch, and rank, and more complex loss components are gradually enabled as training progresses.

5. Datasets, implementation, and empirical performance

Evaluation is reported on two datasets (Yuan et al., 3 Oct 2025). The Combined Brain Tumor Dataset merges BraTS2019, BraTS-MEN, and BraTS-MET and contains 2,547 patients, comprising 795 glioma, 1,424 meningioma, and 328 metastasis cases, each with T1-weighted (T1n), T1-contrast enhanced (T1c), T2-weighted (T2w), and FLAIR (T2f) sequences. The ISLES 2022 dataset contains 400 multi-center MRI cases with DWI, ADC, and FLAIR. All slices are resampled to $_2$8, and augmentation includes horizontal flipping with probability $_2$9, mild color jitter, Gaussian blur with probability $d^0$0, random resized cropping with scale $d^0$1–$d^0$2, and normalization.

Implementation uses PyTorch on NVIDIA A100 hardware. The SAM2 encoder is fine-tuned with LoRA rank $d^0$3. Training uses batch size 48 with gradient accumulation, runs for 100 epochs, and adopts a cosine learning-rate schedule starting at $d^0$4, with a slightly higher learning rate for more target modalities. Comparisons are made against MM-Synthesis, pGAN, and MM-Transformer.

Across multiple modality combinations, the paper reports that Med-K2N achieves the best PSNR/SSIM values, and that all statistical comparisons to Med-K2N are significant at $d^0$5 under the Wilcoxon signed-rank test. Representative results are shown below.

Setting	Best prior baseline	Med-K2N
Brain tumor, input T1n only	MM-Transformer: 28.89 / 0.941	29.46 / 0.947
Brain tumor, input T1n + T2w	MM-Transformer: 29.21 / 0.944	29.78 / 0.949
Brain tumor, input T1n + T2w + T2f	MM-Transformer: 29.95 / 0.950	30.58 / 0.955
ISLES 2022, input ADC only	MM-Transformer: 25.62 / 0.915	26.72 / 0.931
ISLES 2022, input FLAIR only	MM-Transformer: 26.21 / 0.926	27.79 / 0.942
ISLES 2022, input ADC + FLAIR	MM-Transformer: 26.54 / 0.932	27.65 / 0.948

Qualitative figures are described as showing progressive improvement in structural fidelity, lesion delineation, contrast consistency, and PSNR as additional modalities become available. This suggests that the sequential fusion mechanism is not merely tolerant of extra modalities but able to exploit them incrementally.

The ablation study isolates the contributions of the main modules on the task of generating T2f from T1n, T1c, and T2w. Baseline fusion (B0) yields 26.53 PSNR and 0.878 SSIM. Adding PreWeightNet (B1) raises performance to 27.05 and 0.895, ThresholdNet (B2) to 27.21 and 0.902, EffiWeightNet (B3) to 27.89 and 0.919, CMIM interaction (B4) to 28.28 and 0.929, and curriculum learning (B5) to 28.41 and 0.933. The largest single improvement in the sequence comes from EffiWeightNet, while CMIM improves quality without extra inference overhead by enforcing modality-consistent representations.

6. Flexibility, clinical relevance, and limitations

A defining property of Med-K2N is that it supports arbitrary subsets of modalities as inputs and arbitrary sets of target modalities as outputs within one model (Yuan et al., 3 Oct 2025). This is enabled by processing each available modality independently into a shared feature space, by conditioning the fusion modules on modality and task embeddings rather than fixed indices, and by using task-specific embeddings and heads in TaskHeadNet. If a modality is missing, it is simply absent from the input set. If an input is noisy or unhelpful, PreWeightNet can assign it a low $d^0$6, ThresholdNet can learn a high $d^0$7, and EffiWeightNet can converge to a low $d^0$8. The quality feedback loop from TaskHeadNet further updates these decisions according to actual output quality.

The clinical applications named in the paper include synthesizing missing MR sequences for brain tumors, generating DWI- or ADC-like maps for stroke imaging, MRI-to-CT synthesis for spine or brain surgery planning, and completing partially acquired studies in resource-limited settings. The anticipated operational benefits are reduced contrast-agent use, reduced radiation exposure, shorter scans, less patient motion, and improved throughput. Because the model is formulated as $d^0$9 rather than as a fixed mapping, it is intended to adapt to varying scanner protocols and institutional acquisition policies.

The stated limitations are equally explicit. Computational complexity rises with the number of input modalities, which may affect real-time deployment. Evaluation is limited to brain imaging, specifically tumors and stroke, so generalization to other organs and modality families remains unvalidated. Domain generalization across new scanners, institutions, and rare modalities is not yet fully explored. The authors also emphasize ethical and safety constraints: synthetic images require careful clinical validation and regulatory review, Med-K2N is intended as a clinical decision support tool rather than a replacement for standard imaging, and use outside the validated scope may pose risks. Future directions named in the paper include lighter-weight architectures, uncertainty quantification, and broader clinical validation studies.

In the literature of medical image synthesis, Med-K2N is therefore best understood as a quality-aware, progressive, and modality-identity-constrained framework for flexible $F_i \in \mathbb{R}^{H \times W \times D}$0 generation. Its distinctiveness lies not in a single component but in the coupling of sequential multimodal processing, pairwise adaptive fusion, and causal modality identity supervision within a unified architecture (Yuan et al., 3 Oct 2025).