Understanding Generation Gap

• Understanding-generation gap is a phenomenon where models display strong semantic or visual understanding yet fail to generate outputs with matching fidelity.
• Key causes include training pipeline bias, representation mismatch, and decoupled optimization, which affect time-series, image, and video generation.
• Bridging strategies involve fidelity-preserving shared representations, understanding-guided generation, and paired evaluation benchmarks to align generation with understanding.

Searching arXiv for papers on the "understanding–generation gap" in multimodal and time-series modeling. Search query: "understanding generation gap unified multimodal models" The understanding–generation gap denotes a recurrent asymmetry in modern multimodal and sequence models: the model can often understand, reason, or describe an input more reliably than it can generate an output that is equally faithful to the same semantics, constraints, or latent structure. In the literature, this gap is instantiated in several closely related ways: as the divide between semantic / textual understanding and high-fidelity numerical generation in time series; as an internal imbalance in unified multimodal models where image understanding is stronger than image generation; as the mismatch between high visual quality and weak semantic or physical grounding in image-to-video and chart generation; and as the discrepancy between instruction understanding, intention reasoning, and reliable generation in language systems (Guan et al., 19 Feb 2026, Pan et al., 6 Mar 2026, Chang et al., 13 Feb 2025).

1. Definition and conceptual scope

In time-series modeling, the gap is defined as the divide between time series understanding and time series generation. Understanding-oriented models optimize for semantic tasks such as question answering, temporal reasoning, explanation, and causal or structural analysis, but struggle with high-fidelity numerical generation; generation-oriented models optimize for forecasting and imputation, but usually lack explicit semantic understanding (Guan et al., 19 Feb 2026). HermesFlow gives an explicit scalar formulation of the same idea:

$$\text{Gap} = \text{Understanding Score} - \text{Generation Score},$$

where understanding is measured by question answering on real images and generation is measured by question answering on generated images for the same underlying semantic content (Yang et al., 17 Feb 2025).

In unified multimodal models, the gap is framed as a systematic, internal capability imbalance: the same model is much better at visual understanding $(I \rightarrow T)$ than at image generation $(T \rightarrow I)$. The understanding branch can recognize fine-grained attributes, track spatial relations, and handle counting and compositional queries reasonably well, while the generation branch often misses objects or attributes, breaks spatial layouts, and fails on counting and combinatorial constraints (Pan et al., 6 Mar 2026). A closely related formulation treats the gap as the discrepancy between what the unified model could correctly compute or recall in its understanding module and what it actually expresses in the generated image (Niu et al., 25 Nov 2025).

The notion also extends beyond images. In LLMs, the “gap between LLMs and human intentions” is decomposed into instruction understanding, intention reasoning, and reliable generation. This makes the understanding–generation gap a pipeline-level phenomenon: failures in instruction parsing or intention inference propagate into unstable, unethical, or factually unreliable outputs (Chang et al., 13 Feb 2025). In image-to-video generation, the gap appears when a model produces visually impressive clips but fails to understand which subject in the input image should move, which attributes should remain bound, or which physical and commonsense consequences should follow (Zhang et al., 29 Sep 2025).

A common misconception is that a shared backbone automatically yields unification. Several studies explicitly reject this view: shared parameters alone do not guarantee that understanding informs generation, that generation improves understanding, or that both branches remain internally consistent (Niu et al., 25 Nov 2025, Liu et al., 25 Jun 2026, Rao et al., 17 Mar 2026).

2. Structural causes

A first major cause is training pipeline bias. Unified multimodal models are typically exposed to massive understanding supervision—captioning, VQA, instruction tuning—but much less direct supervision on instruction-following text-to-image generation for complex prompts. The shared backbone therefore becomes a strong visual/textual encoder-decoder, while the generation head remains relatively under-optimized (Pan et al., 6 Mar 2026). In language systems, an analogous imbalance appears because long-context instruction following and multi-turn conversational state tracking are harder to preserve than locally fluent generation (Chang et al., 13 Feb 2025).

A second cause is representation mismatch. Several papers argue that understanding and generation require fundamentally different representational granularities. Reconstruction-oriented tokenizers capture low-level perceptual details and are suitable for generation, but lack high-level semantics; contrastive encoders align well with language and support understanding, but cannot decode reliably back to pixels (Song et al., 18 Mar 2025, Qu et al., 2024). DualToken and TokenFlow both identify the same conflict: forcing one codebook or one visual vocabulary to serve both semantics and reconstruction degrades both sides. This suggests that the gap is partly a consequence of representational overload rather than only insufficient scale.

A third cause is branch decoupling and optimization asymmetry. In hybrid unified models, the understanding branch and the generation branch may share a backbone while their downstream objectives remain largely independent. “Does understanding truly inform generation?” is therefore not a rhetorical question but an architectural one (Niu et al., 25 Nov 2025). The DPO study on Janus-Pro gives a particularly sharp diagnosis: understanding and generation gradients are near-orthogonal with ~11–14x magnitude imbalance driven by VQ token count asymmetry, and generation quality resists DPO alignment across all tested conditions on that architecture (Rao et al., 17 Mar 2026). Here the gap is not only behavioral but gradient-level.

A fourth cause is task intrinsic difficulty. Recognition maps a rich visual signal to a more abstract description; constrained synthesis must realize a compact semantic specification as a complex visual or temporal configuration. The GvU paper treats this as an intrinsic asymmetry: standard generative losses do not explicitly reward semantic fidelity to text, especially for colors, counts, relations, and compositional constraints (Pan et al., 6 Mar 2026). In time series, the asymmetry appears when text-native tokenization breaks numerical continuity, making long, precise numeric generation difficult even for models that can answer discrete semantic questions (Guan et al., 19 Feb 2026).

A fifth cause is missing structural or physical world models. GOBench shows that current multimodal LLMs can produce images that look optically plausible and aesthetically pleasing, yet perform poorly at reasoning about the same geometric-optics phenomena those images depict. The paper attributes this to texture/appearance bias, lack of explicit physics supervision, and vision encoders that do not encode geometry explicitly (Zhu et al., 1 Jun 2025). UI2V-Bench makes a parallel point for video: diffusion models are trained to denoise and reconstruct frames, but their objectives do not explicitly enforce instance-level grounding, physical laws, or temporal causality beyond short-term consistency (Zhang et al., 29 Sep 2025).

3. Manifestations across domains

The gap is not confined to a single modality or benchmark family. It appears wherever semantic fidelity, structured reasoning, and controllable synthesis must coexist.

Domain	Stronger side	Weaker side
Time series	semantic / textual understanding	high-fidelity numerical generation
Unified image models	visual understanding	text-to-image generation
Geometric optics	visually plausible image generation	optical understanding
Image-to-video	video quality and motion smoothness	semantic understanding and reasoning
Cinematography	recognition of shot scale, angle, color	faithful camera movement generation
Infographic charts	chart reading after fine-tuning	infographic chart code generation

TimeOmni-VL states the divide directly: understanding lives in the semantic / textual space, while generation lives in the numeric / visual space. The paper reports that base Bagel scores 0.0 on QA1 and QA5, while TimeOmni-VL reaches 1.0 on QA1, 1.0 on QA2, 0.931 on QA3, 1.0 on QA4, 0.667 on QA5, and 0.841 on QA6; at the same time it remains competitive on forecasting and achieves state-of-the-art imputation (Guan et al., 19 Feb 2026). This is an explicit attempt to collapse the divide rather than optimize one side only.

GvU frames the same phenomenon inside unified multimodal models: base X-Omni scores 0.68 on GenEval overall, while GvU reaches 0.81 without prompt rewriting and 0.84 with prompt rewriting; on GenEval++ the score rises from 0.282 to 0.404. Understanding remains comparable on POPE, GQA, MMB, SEED, DocVQA, and OCRBench, while some MMT-Bench fine-grained categories improve (Pan et al., 6 Mar 2026). The gap narrows, but is not eliminated.

GOBench reveals a striking cross-modal asymmetry. On generation, GPT‑4o‑Image reaches 4.09 Optical Authenticity, 3.95 Aesthetic Quality, and 3.97 Instruction Fidelity, while Seedream 3.0 reaches 4.02 Optical Authenticity and 4.02 Aesthetic Quality. On understanding, the best Optical Authenticity SPS is only 37.35% for Gemini‑2.5 Pro, with Claude 3 Opus at 33.93% and GPT‑4o at 32.63% (Zhu et al., 1 Jun 2025). The core pattern is that models know that a scene “should” contain shadows, reflections, or refractions, but not reliably whether those effects obey geometric-optics constraints.

UI2V-Bench shows a comparable split in image-to-video systems. All tested models have Image Quality ~0.71–0.73, Motion Smoothness ~0.986–0.994, and Video–Image Similarity ~0.89–0.93, yet their Image Understanding scores range from 28.49 to 46.30. Hailuo reaches 46.30, SeedDance 41.67, Wan2.1 38.68, CogVideoX 32.93, and HunyuanVideo 28.49 (Zhang et al., 29 Sep 2025). High-quality motion therefore does not imply correct subject grounding, attribute binding, or reasoning.

CineTechBench exposes the same problem in cinematography. On image QA, GPT‑4o reaches 70.16% overall and Gemini‑2.5‑Pro 69.67%, with color being very strong and focal length being the weakest dimension; on video QA for camera movement, Gemini‑2.5‑Pro reaches 56.69% overall, and rotation remains difficult. On generation, Kling v1.6 achieves the best RotErr: 21.68, TransErr Rel: 48.49, CamMC Rel: 62.57, and CLIP‑IS: 90.15, yet qualitative failures still include omitted roll and reversed roll direction (Wang et al., 21 May 2025). This suggests that categorical recognition of cinematographic technique is easier than faithful continuous camera trajectory control.

HermesFlow makes the gap fully explicit on homologous image–text data. For VILA-U (7B) the gap is 0.169, for Janus (1.3B) 0.182, for Show-o (1.3B) 0.087, and for HermesFlow (1.3B) 0.036 (Yang et al., 17 Feb 2025). Self-Contradiction as Self-Improvement reaches a similar conclusion from the opposite direction: the self-contradiction mainly arises from weak generation that fails to align with prompts, rather than misunderstanding (Han et al., 22 Jul 2025).

4. Strategies for bridging the gap

One strategy is shared representation with explicit fidelity guarantees. TimeOmni-VL introduces Bi-TSI, a fidelity-preserving bidirectional mapping between time series and images, so that understanding tasks become image understanding and generation becomes image editing or inpainting. The model then uses understanding as an explicit conditioning signal for diffusion-based forecasting and imputation (Guan et al., 19 Feb 2026). A related representational strategy appears in DualToken and TokenFlow: both use dual codebooks or dual visual vocabularies to decouple high-level semantic features from low-level perceptual features while maintaining alignment via a shared mapping mechanism (Song et al., 18 Mar 2025, Qu et al., 2024).

A second strategy is understanding-guided generation. GvU turns the understanding branch into an internal evaluator and defines a token-level intrinsic reward

$R_i(T, I_i) = P(T \mid I_i),$

which is then used in GRPO to update the generative policy (Pan et al., 6 Mar 2026). TimeOmni-VL uses a calibrated Chain-of-Thought as an explicit plan for generation; removing CoT conditioning causes an 8.2% increase in nMASE on average (Guan et al., 19 Feb 2026). UniSandbox similarly shows that explicit CoT in the understanding module effectively bridges the gap on reasoning generation tasks, and that self-training can internalize this ability (Niu et al., 25 Nov 2025).

A third strategy is paired or contradiction-based post-training. HermesFlow constructs homologous preference data and optimizes a Pair-DPO objective over understanding and generation jointly, so that both are aligned around the same semantic core (Yang et al., 17 Feb 2025). Self-Contradiction as Self-Improvement defines the Nonunified score,

$$\text{Nonunified score} := \frac{1}{|\mathcal{Y}|} \sum_{\mathbf{y} \in \mathcal{Y}} \mathbb{I}\left[ \pi^{\text{und}}_{\theta}(\mathbf{x}, q(\mathbf{y})) \neq 1 \right], \quad \mathbf{x}=\pi^{\text{gen}}_{\theta}(\mathbf{y}),$$

and then uses self-contradiction itself as supervision for SFT and DPO (Han et al., 22 Jul 2025).

A fourth strategy is adapter-based modality bridging. Omni-Video augments a video MLLM with a vision head that produces continuous visual clues and an adapter that maps them into the conditional space of a diffusion decoder. This creates a trainable bridge from multimodal understanding to video generation and instruction-based editing without retraining the entire system from scratch (Tan et al., 8 Jul 2025).

A fifth strategy is benchmark-coupled and decoupled evaluation. Unison organizes unified evaluation into internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement, while also providing unified and decoupled tracks for failure attribution (Liu et al., 25 Jun 2026). ChartGalaxy, GOBench, UI2V-Bench, and CineTechBench each do something similar within their own domains: they make the gap measurable rather than anecdotal (Li et al., 24 May 2025, Zhu et al., 1 Jun 2025, Zhang et al., 29 Sep 2025, Wang et al., 21 May 2025).

5. Evaluation paradigms and diagnostic evidence

A central lesson of the recent literature is that isolated metrics hide the gap. Unison therefore defines explicit joint scores. For internal consistency,

$$\text{Score}_\text{ic} = \frac{1}{n} \sum_{i=1}^{n} f(a_i, s_i^{und}, s_i^{gen}),$$

where both the understanding answer on the real image and the judge’s answer on the generated image must agree with the ground-truth attribute (Liu et al., 25 Jun 2026). For understanding-guided generation, the benchmark combines localization and editing:

$$\text{Score}_\text{ugg} = \frac{ IoU(\mathcal{R},\mathcal{R}_{gt}) + \mathcal{J}(I, I_{ie}) }{2}.$$

This makes it possible to separate “the model can localize” from “the model can edit” and from “the localization actually guides the edit” (Liu et al., 25 Jun 2026).

TimeOmni-VL couples six understanding tasks with two generation tasks on the same instances, precisely so that improvements in one side can be tested for consequences on the other. The paper reports that replacing their TS2I with a generic heatmap causes forecasting and imputation nMASE to increase substantially, while freezing the understanding model and removing CoT conditioning causes generation nMASE to increase by 8.2% on average (Guan et al., 19 Feb 2026). This is unusually direct evidence that better understanding can improve numerical generation rather than merely post-hoc interpretability.

The DPO study of Janus-Pro is equally diagnostic in the opposite direction. No method improves generation CLIPScore at 7B; at 1B, all methods degrade generation. Gradient analysis shows cos ~ 0 between understanding and generation gradients, with ~11–14x magnitude imbalance driven by 576 generation tokens vs. ~30–100 text tokens. The generation DPO loss converges to $\ln(2)$, indicating no meaningful separation between preferred and dispreferred sequences (Rao et al., 17 Mar 2026). This result directly challenges the assumption that a generic alignment objective can bridge the gap on a VQ-based unified architecture.

ChartGalaxy extends diagnosis to structured infographic generation. Its benchmark combines a high-level GPT‑4o visual similarity score with a low-level SVG structural matching score over Area, Text, Image, Color, Position, and Size. The best proprietary LVLM, Gemini‑2.5‑Pro, reaches an overall score of 85.21, while the best open-source LVLM, Llama-4-Maverick-17B, reaches 61.29 (Li et al., 24 May 2025). The metric design makes a crucial point: generation errors are often structural rather than merely perceptual.

A further misconception is that intrinsic consistency metrics alone are sufficient. Self-Contradiction as Self-Improvement shows that Nonunified score cannot distinguish co-degradation from co-improvement: under corrupted supervision, generation and understanding can both worsen while internal consistency appears to improve (Han et al., 22 Jul 2025). This suggests that external checks, trusted judges, or strict data quality controls remain necessary.

6. Limitations and open research problems

Several limitations recur across domains. Residual asymmetry remains even after explicit bridging. GvU improves generation substantially, but the paper still states that the gap is reduced rather than eliminated, especially on complex multi-object, multi-relation prompts (Pan et al., 6 Mar 2026). UniSandbox reaches strong gains with explicit and implicit CoT, yet still treats its synthetic tasks as a preliminary tool rather than a full account of real-world reasoning generation (Niu et al., 25 Nov 2025).

Representation dependence remains a major constraint. TimeOmni-VL depends on known periodicity and image-based encoding, and its high-resolution TS-images impose substantial token budgets (Guan et al., 19 Feb 2026). The DPO study suggests that discrete VQ tokenization is a structural bottleneck for generation alignment under offline preference optimization (Rao et al., 17 Mar 2026). Omni-Video reduces the video understanding–generation gap, but still relies on a bridged rather than fully end-to-end-shared diffusion–MLLM interface (Tan et al., 8 Jul 2025).

Grounding to physical, causal, or domain-specific structure remains incomplete. GOBench shows that models still lack reliable optical understanding even when images appear plausible (Zhu et al., 1 Jun 2025). UI2V-Bench shows that image-to-video systems remain weak on category understanding and on reasoning categories such as Physical Interactions and Temporal Changes (Zhang et al., 29 Sep 2025). CineTechBench shows that camera movement generation is far from cinema-grade even when static understanding is moderate to strong (Wang et al., 21 May 2025). This suggests that scaling and better prompts alone are unlikely to close the gap in physically governed domains.

Evaluation itself is still an open problem. Unison-Judge, ChartGalaxy’s SVG metrics, and Nonunified each illuminate different parts of the problem, but no single metric currently captures semantic fidelity, controllability, internal consistency, and mutual enhancement simultaneously (Liu et al., 25 Jun 2026, Li et al., 24 May 2025, Han et al., 22 Jul 2025). A plausible implication is that future progress will depend on paired benchmark design: joint tasks, decoupled attribution, and external validation must be combined.

Taken together, the literature presents the understanding–generation gap not as a single bug but as a structural property of current model design. Shared backbones, larger datasets, or stronger decoders do not by themselves guarantee that understanding informs generation. The strongest results instead come from systems that make the connection explicit: fidelity-preserving shared representations, understanding-guided generation, internal evaluators or preference signals, curriculum-based self-improvement, and benchmarks that test not only whether a model can understand and generate, but whether the two capabilities are actually the same capability expressed in opposite directions (Guan et al., 19 Feb 2026, Pan et al., 6 Mar 2026, Yang et al., 17 Feb 2025, Liu et al., 25 Jun 2026).