SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment

Abstract: Unifying multimodal understanding and generation is a compelling frontier that is beginning to emerge in the medical field. However, the limited existing unified medical models typically treat understanding and generation as disjoint objectives, lacking a meaningful functional synergy. In this work, we identify and address a critical question in unified medical modeling: what form of understanding truly benefits generation. We present SynerMedGen, a unified framework built on the proposed principle of generation-aligned understanding, which synergizes understanding objectives with generation tasks via task alignment. SynerMedGen introduces three generation-aligned understanding tasks and a two-stage training strategy that transfers generation-beneficial representations learned during understanding training to medical image synthesis. Remarkably, even with understanding training alone, our SynerMedGen achieves strong zero-shot performance across 22 medical image synthesis tasks and demonstrates robust generalization to unseen datasets. When combined with generation training, SynerMedGen consistently outperforms state-of-the-art specialized medical image synthesis models as well as recent unified medical models. We also release a large-scale dataset named SynerMed consisting of 1M paired synthesis samples and 2M generation-derived understanding instances to support further research on understanding-generation synergy. Our project can be accessed at https://github.com/Mhilab/SynerMedGen.

• The paper introduces generation-aligned understanding (GAU), using conditional target selection, modality identification, and transformation instruction alignment to train representations directly from paired synthesis data.
• SynerMedGen combines GAU pretraining with flow-matching generation and outperforms unified and specialized baselines across 22 tasks, including 92.45% SSIM on BraTS T2→T1CE.
• The method shows strong zero-shot transfer before generation training and generalizes to unseen datasets, while its clinical validity, 3D applicability, and broader task-derivation strategy remain open questions.

Motivation and problem statement

Unified medical multimodal LLMs (MLLMs) that couple visual understanding with image generation have begun to appear, exemplified by HealthGPT and UniMedVL. The authors of SynerMedGen (2605.08724) argue that these systems share a structural weakness: the understanding branch is trained with proxy objectives (e.g., recognition-style VQA) that are only weakly related to what generation actually requires. For cross-modality medical synthesis—arguably the most clinically consequential generation task in medicine—the model must preserve slice-level patient-specific anatomy, treat modality as an explicit control factor, and respect the direction of the intended transformation. Generic semantic supervision provides none of these guarantees, producing a task misalignment in which a model can improve at answering questions while its representations remain useless for synthesis.

The paper's central claim is that this misalignment, not architecture or scale, is the bottleneck in unified medical modeling. Its proposed remedy is deliberately simple: derive understanding tasks directly from the paired generation dataset itself, so that understanding supervision is co-defined with generation objectives—a principle the authors call generation-aligned understanding (GAU).

Method

SynerMedGen builds on the Bagel unified architecture, using a dual-encoder design: a ViT encoder produces semantic tokens for the understanding expert, while a VAE encoder produces latent tokens for the generation expert, both projected into a shared Mixture-of-Transformer-experts hidden space. The authors note the design principle is architecture-agnostic.

Three generation-aligned understanding tasks are constructed from each paired synthesis sample:

• Conditional Target Selection (CTS): given a source slice and target modality request, select the correctly paired target slice among $N$ candidates, with hard negatives drawn from neighboring slices ($\delta \in \{\pm1,\ldots,\pm K\}$) to force fine-grained rather than coarse-semantic alignment.
• Modality Identification (MI): predict the input modality (CT, CBCT, PET, MRI sequences), including confusable pairs such as CT vs. CBCT, making modality an explicit learnable control factor.
• Transformation Instruction Alignment (TIA): match an observed paired change to the correct route-level description among distractors including swapped directions, grounding what changes (appearance/contrast) versus what must remain invariant (structures/lesions).

All three are trained as prompted next-token prediction with cross-entropy restricted to answer tokens. Training proceeds in two stages: Stage I (GAU) learns the aligned representation on the understanding pathway; Stage II (UCG) trains conditional synthesis via flow matching in VAE latent space, predicting the velocity field $v_\theta(z_t,t,c)$ conditioned on source evidence, initialized from the Stage I weights.

The authors release SynerMed, a dataset integrating BraTS, SynthRAD2023, and AutoPET into 1M paired samples spanning 22 directed synthesis tasks, plus roughly 2M derived understanding instances.

Main results

Across all 22 cross-modality synthesis tasks on SynthRAD2023, AutoPET, and BraTS, SynerMedGen outperforms both unified baselines (HealthGPT, UniMedVL) and specialized synthesis models (Pix2Pix, CycleGAN, BBDM, ResViT, SynDiff, RCD) on SSIM, PSNR, and MAE. Representative SSIM results:

Task	Best specialized baseline	HealthGPT	UniMedVL	SynerMedGen
Brain CBCT→CT	85.97 (RCD)	57.37	51.48	87.15
Whole-body CT→PET	88.90 (RCD)	66.54	74.12	91.10
BraTS T2→T1CE	88.15 (SynDiff)	67.13	73.44	92.45
BraTS T2→T1	88.31 (SynDiff)	60.13	77.26	90.58

On BraTS T2→T1CE the improvement over SynDiff is 4.3% SSIM; MAE and PSNR comparisons show consistent but smaller margins (e.g., PSNR gains of +0.59 to +7.19). Notably, the unified baselines HealthGPT and UniMedVL frequently underperform even Pix2Pix on CBCT↔CT tasks, which itself substantiates the paper's claim that current unified medical models do not produce generation-beneficial representations.

Evidence that task alignment, not extra training, drives the gains

The most informative experiments isolate the contribution of GAU under strictly controlled conditions—identical architecture, data, schedule, and Stage II training—with only the understanding tasks swapped between GAU and HealthGPT's traditional supervision. Three findings stand out:

1. Zero-shot transfer after Stage I alone: without any generation training, GAU-pretrained models already synthesize strongly (e.g., +41.72% SSIM on BraTS T1→T2, +62.94% on whole-body PET→CT relative to the Bagel baseline), whereas traditional understanding yields limited, inconsistent improvements. This is the paper's strongest claim: understanding supervision alone, if properly aligned, transfers to synthesis zero-shot.
2. Stage II comparison: traditional understanding can even degrade certain routes (e.g., T1→FLAIR), while GAU initialization consistently improves final performance (up to +9.28% SSIM on FLAIR→T1).
3. Understanding-task accuracy: SynerMedGen achieves the highest accuracy on CTS, MI, and TIA evaluations, supporting the interpretation that gains stem from improved shared representations rather than incidental factors.

Generalization results reinforce this. On the unseen MyoPS cardiac MRI dataset (unseen by all evaluated methods), SynerMedGen-GAU substantially outperforms Bagel, HealthGPT, and UniMedVL in SSIM and MAE; on unseen SynthRAD2025 (new organs, same modalities), the full model beats SynerMedGen-UCG and both unified baselines. Ablating the three tasks progressively (CTS → +MI → +TIA) shows monotonic improvement, indicating complementary contributions. A sensitivity study on the CTS hard-negative window finds $K=5$ optimal; larger windows degrade performance by diluting the correspondence signal with easy negatives.

Limitations and open questions

Several caveats deserve emphasis. First, HealthGPT and UniMedVL were evaluated via public checkpoints rather than retraining, so part of their deficit could reflect checkpoint quality or scale differences rather than purely task misalignment—although the controlled swap experiment against traditional understanding mitigates this concern for the core claim. Second, the framework is instantiated and validated only on cross-modality slice-level 2D synthesis; whether GAU transfers to other generation tasks (segmentation-conditioned synthesis, 3D volumes, report-grounded generation) remains untested, as the authors acknowledge. Third, the three tasks are hand-designed around the specific requirements of modality translation; a general procedure for automatically deriving aligned tasks from arbitrary generation datasets is not provided. Fourth, evaluation relies on pixel-level metrics (SSIM, PSNR, MAE); clinical validity measures such as downstream diagnostic accuracy or lesion-detection fidelity are not reported. Finally, the two-stage schedule assumes Stage I representations remain compatible with flow-matching training—an assumption supported empirically here but not analyzed mechanistically.

Conclusion

SynerMedGen makes a focused argument: in unified medical MLLMs, the form of understanding supervision matters more than the fact of having an understanding branch at all. By constructing CTS, MI, and TIA directly from paired synthesis data and transferring the resulting representations through a two-stage flow-matching pipeline, the method achieves state-of-the-art results across 22 synthesis tasks, strong zero-shot synthesis before any generation training, and generalization to unseen datasets and modalities. The released SynerMed corpus (1M paired samples, ~2M aligned understanding instances) provides a concrete testbed for further work on understanding–generation synergy, and the central open question—whether generation-aligned task derivation generalizes beyond cross-modality translation—remains open.