Transferability Between Understanding and Generation in Unified Multimodal Models

Abstract: Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

• The paper introduces capability-level transferability tests showing that knowledge learned for image understanding can improve generation, especially in models with shared visual and language representations.
• The experiments find that Lumina-DiMOO achieves the strongest bidirectional transfer, while understanding-to-generation training improves counting, spatial reasoning, and text rendering with less image-quality degradation than direct generation fine-tuning.
• The results show that transfer direction depends on visual granularity: understanding supervision suits structural skills such as counting, whereas generation supervision better supports pixel-precise abilities such as text rendering.

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, but the mechanisms through which the two tasks interact remain poorly characterized. The paper under review, "Transferability Between Understanding and Generation in Unified Multimodal Models" (2607.04423), addresses this gap by introducing transferability—whether training a specific capability (e.g., counting) on one task improves the same capability on the other without explicit supervision—as a direct, capability-level probe of cross-task interaction, in contrast to prior work that infers synergy from aggregate benchmark movements.

Motivation and positioning

Existing studies of understanding–generation interaction in UMMs typically report that adding generation training improves understanding benchmarks (e.g., MMMU, POPE) or vice versa (2607.04423). The authors argue that such aggregate observations cannot distinguish genuine cross-task knowledge transfer from confounds such as additional data or regularization. Their intervention is deliberately narrow: fine-tune on a single capability in one task, evaluate that same capability in the other task, under controlled LoRA-based supervised fine-tuning on 7B–8B open-source models.

Architecture-dependent transfer

Using counting as the probe capability, the authors evaluate four representative architectures spanning the main UMM design families: Lumina-DiMOO (shared transformer with unified image encoder), Janus-Pro (shared transformer with separate encoders), BAGEL (mixture-of-transformers with joint attention), and BLIP3-o (decoupled transformers with a diffusion module). Training data are derived from PixMo-Count and PixMo-Points (counts 0–20), with matched generation prompts produced by Qwen3-VL captioning.

The central empirical finding is that transferability is architecture-dependent. Understanding-to-generation transfer ($\text{und}\rightarrow\text{gen}$) improves counting generation in all models except Janus-Pro (e.g., Lumina-DiMOO: 48.0%→57.0% accuracy, MAD 1.11→0.83), whereas generation-to-understanding transfer ($\text{gen}\rightarrow\text{und}$) appears only in Lumina-DiMOO and BAGEL. Lumina-DiMOO, the fully shared-backbone model with a unified visual encoder, exhibits the strongest bidirectional transfer. The authors attribute this to the degree of representation sharing: when both tasks operate over a common representation space, knowledge acquired under one objective propagates to the other, whereas decoupled visual pathways (Janus-Pro, BLIP3-o) offer fewer such opportunities. An appendix experiment on MMaDA, an independently trained model of the same architectural family, shows consistent bidirectional gains (e.g., spatial-relation generation accuracy 33.0%→46.3%), supporting the claim that the effect tracks the architecture rather than one particular checkpoint—though the authors concede that broader coverage across model families is limited by computational cost and code availability.

Transfer as a practical training strategy

The second contribution exploits $\text{und}\rightarrow\text{gen}$ transfer as an alternative to direct generation fine-tuning. Direct fine-tuning on skill-focused generation data induces distribution shift that degrades visual quality—a concern amplified for models whose generation distributions have been carefully aligned via preference-based post-training. The authors compare the two strategies on Lumina-DiMOO across three capabilities:

• Counting: understanding-based training matches direct generation training on accuracy (57.0% for both) but achieves a lower MAD (0.83 vs. 0.91) while preserving image quality (IS 17.55 vs. 15.29; FID 31.47 vs. 52.51 relative to baseline outputs).
• Spatial relation: using a synthetic 200K-sample diagonal-relation dataset built from ImageNet, transfer yields +7.0% accuracy versus +13.0% for direct training, with lower distribution shift (FID 30.95 vs. 32.28); direct training produces images resembling pasted crops on canvases.
• Text rendering: using a 200K synthetic dataset of rendered Markdown documents, transfer improves most OCR-based metrics (WER 0.654→0.641, METEOR 0.403→0.410) with minimal quality loss, while direct generation training yields larger metric gains but degrades quality.

Across all three tasks, transfer-based training is the safer option: it attains a substantial fraction of direct training's capability gains while keeping FID and IS near baseline. The authors also verify on POPE, MMBench, MMMU, and MME that understanding-side capability training does not degrade general multimodal understanding—MMMU in fact improves slightly (57.7→62.0 after counting training).

Capability-dependent transfer direction

A notable secondary finding is that the dominant transfer direction is capability-dependent. Quantifying transfer strength as the ratio of transfer-induced improvement to direct-training improvement, the authors find that $\text{und}\rightarrow\text{gen}$ is stronger for counting (100% of direct-training gain, and 140% for MAD) and spatial relation (53.8%), but weak for text rendering (e.g., 7.6% on F1). The reverse, $\text{gen}\rightarrow\text{und}$, is comparatively more effective for text recognition (e.g., 25.9% on F1 vs. 7.6%). Their explanation is granularity-based: counting and spatial layout are high-level structural concepts learnable through understanding supervision, whereas glyph rendering requires pixel-level precision that understanding objectives—known to exploit contextual shortcuts—fail to encode, but which generation supervision forces. This suggests a practical heuristic: choose the transfer direction according to the visual granularity the target capability demands, although the paper offers no principled method for predicting this in advance.

Joint training and additional analyses

The paper also examines whether joint und+gen training on the same capability data yields simultaneous gains; it does not. Performance oscillates over training epochs—understanding accuracy rises while generation accuracy falls, then the trend reverses—consistent with conflicting optimization signals between tasks, echoing UniFork's finding that ideal representations differ across layers. The authors are careful to note that this result may be an artifact of their setting: post-training on a pretrained model with tightly paired, concept-focused data differs from large-scale pre-training, and distinguishing between these explanations remains an open question. Supplementary analyses include an attention-level visualization showing that transferred understanding training sharpens text-to-image attention in Lumina-DiMOO but not in Janus-Pro, a LoRA-rank ablation showing transfer strength grows monotonically with adaptation capacity (accuracy 48.0%→53.6% from rank 8 to 128, with full fine-tuning collapsing generation quality), and a preliminary composition experiment in which a model trained on a mixture of counting and spatial-relation understanding data improves joint accuracy on unseen compositional prompts (24.7%→28.5%), indicating transferred capabilities can be composed at generation time.

Limitations and open questions

The paper's coverage is limited to four representative models, one per architectural family, so the architecture-transferability claim rests on a small sample despite the MMaDA corroboration. Transferability also cannot be attributed to architecture alone—its direction and magnitude depend on the capability's required visual granularity, and no method is given to anticipate the optimal transfer direction a priori. The joint-training result is confounded by the post-training setting and the tightly paired data distribution. Whether the observed transfer dynamics would hold under large-scale pre-training, and how to predict transferability across tasks in advance, are left unresolved.

Conclusion

This paper reframes the study of understanding–generation interaction in UMMs at the capability level, establishing through controlled interventions that cross-task transfer exists, is strongest in fully shared architectures with unified visual encoders, and can be exploited as a practical post-training strategy: injecting a target capability via understanding supervision improves generative performance while avoiding the distribution shift that direct generation fine-tuning incurs. The finding that transfer direction is governed by the granularity of required visual knowledge adds a useful design heuristic, and the negative result on joint training tempers expectations that transferability implies simultaneous optimization gains.