Comprehension-Generation Gap

• Comprehension-Generation Gap is a misalignment between a system’s latent understanding and its produced output.
• It manifests in diverse areas such as reading comprehension, multimodal modeling, and software engineering, highlighting mismatches in evaluation and construct validity.
• Bridging the gap requires coupling reasoning with generation to preserve context and ensure that output quality reflects deep comprehension.

The comprehension-generation gap denotes a family of mismatches between understanding and production. In some settings, it is the distance between what a human reader needs in order to understand a text and the kinds of questions standard systems are able to generate. In others, it is the mismatch between a model’s comprehension-side competence and its ability to transfer that competence into generation, editing, or interactive behavior. In software engineering, a closely related formulation describes cases in which generative systems improve task completion while leaving human understanding unchanged. Across these literatures, the common theme is that successful output generation does not by itself establish deep alignment with the underlying comprehension process (Ko et al., 2020, Gul et al., 2024, Wang et al., 2 Feb 2026, Qiao et al., 4 Nov 2025).

1. Core formulations of the gap

The term is not used in a single uniform sense. Instead, recent work applies it to several recurrent mismatches: reader-state versus generated question, diagnostic target versus fluent assessment item, understanding module versus generative pathway, and human productivity versus durable comprehension. These formulations are distinct, but they all treat generation quality as insufficient unless it is grounded in the correct latent understanding or shared context (Ko et al., 2020, Ma et al., 9 Jun 2025, Wang et al., 2 Feb 2026, Qiao et al., 4 Nov 2025).

Domain	Gap formulation	Representative evidence
Reading-time question generation	Reader knowledge gaps versus surface-level factoid questions	INQUISITIVE collects ~19K questions during reading (Ko et al., 2020)
Diagnostic reading assessment	General item quality versus intended inference type	93.8% good-quality, 42.6% target-type match (Ma et al., 9 Jun 2025)
Unified multimodal modeling	Understanding competence versus generative transfer	Persistent text-image gap in GapEval (Wang et al., 2 Feb 2026)
GenAI-assisted programming	Performance gains versus unchanged comprehension	Copilot improved time and tests, not comprehension (Qiao et al., 4 Nov 2025)

A recurring implication is that fluency, benchmark accuracy, or task success can mask a deeper failure of construct alignment. In the assessment setting, this appears as “type drift” toward factual or literal questions. In multimodal systems, it appears as correct textual answers that do not translate into correct images. In programming, it appears as passing more tests without building a stronger mental model of the legacy codebase. This suggests that the gap is best understood as a structural misalignment between what a system can produce and what it actually understands or helps a human understand (Ma et al., 9 Jun 2025, Niu et al., 25 Nov 2025, Ahmad, 14 Apr 2026).

2. Reading-time comprehension and question generation

A direct formulation appears in "Inquisitive Question Generation for High Level Text Comprehension" (Ko et al., 2020). INQUISITIVE contains approximately 19K questions from 1,500 news articles, with 500 each from the Wall Street Journal portion of Penn Treebank, Associated Press articles from TIPSTER, and Newsela at the highest reading level. Questions were elicited sentence by sentence while reading: annotators saw one sentence at a time, could highlight a span, could ask up to 3 questions per sentence, and did not see the next sentence until they were done. A second validation step required questions to be grammatically valid, semantically plausible, related to the highlighted span, and not already answered in prior context; about 5% were filtered out. The split is 15,931 training questions, 1,991 validation questions, and 1,894 test questions. The dataset is explicitly positioned as a response to the comprehension-generation gap because standard factoid QA-style resources train systems to ask answer-extractable questions after reading, whereas readers ask curiosity-driven questions during comprehension (Ko et al., 2020).

Its question distribution emphasizes high-level semantic and discourse processing. In a manually analyzed sample, 38.7% were causal “why” questions or paraphrases, 21.6% elaboration questions, 12.6% definition questions, about 10% background information questions, 8.1% instantiation questions, and 4.5% forward-looking questions. The data also show low lexical overlap and high diversity: only 22% of span words appear in the corresponding question, and the distinct bigram score is 0.41, compared with QuAC at 0.26 and NewsQA at 0.29. A GPT-2-medium model, Inquirer, conditions on article context up to the current sentence, the current sentence, and the highlighted span. Under gold-span conditioning, human evaluation gives Inquirer Valid 0.758, Related 0.887, Answered 0.096, and Salience 3.65, versus Human at Valid 0.958, Related 0.866, Answered 0.047, and Salience 3.18. Span-only input is weakest, broader context improves generation, and SQuAD pretraining does not improve the task, underscoring that reading-time inquisitiveness is not reducible to factoid question generation (Ko et al., 2020).

A different but related formulation appears in "Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment" (Rabin et al., 2023). There the gap is defined between a complete source text $T_C$ and a student text $T_S$, and a gap-focused question is a question whose answer corresponds to information present in $T_C$ but absent from $T_S$. The method is pipeline-based: a constituency parser decomposes $T_C$, a question generator produces candidate questions from spans, a QA model filters out questions already answerable from $T_S$, and the remaining questions are ranked to minimize leakage of unknown information. The desiderata are P1: Answerability, P2: Answers should not be in the common ground, and P3: Questions should only use information known to the user. Using T5-xxl fine-tuned on SQuAD1.1 for question generation, T5-xxl fine-tuned on SQuAD2.0 for QA, and the Berkeley Neural Parser, the system improves from Step 1 at 3.72 to Step 2 at 3.86 to Step 3 at 3.94, with Human at 4.06. The result is not merely fluent questioning, but questioning that respects common ground and targets missing understanding (Rabin et al., 2023).

Open-retrieval conversational MRC introduces yet another variant. In OR-ShARC, the system must decide Yes, No, or Inquire, and if the decision is Inquire it must generate a follow-up question. "Bridging The Gap: Entailment Fused-T5 for Open-retrieval Conversational Machine Reading Comprehension" (Zhang et al., 2022) argues that pipeline designs separate decision-making, span extraction, and rephrasing, so entailment reasoning used for classification is difficult to share with generation. Entailment Fused-T5 addresses this with a shared encoder and duplex decoder: an activated entailment reasoning decoder and a fused answer generation decoder, jointly trained with

$$\mathcal{L} = \mathcal{L}_{answer} + \lambda \mathcal{L}_{entail}.$$

On OR-ShARC, EFT improves dev Micro-Acc from 80.5 to 83.4 and Question generation F1-BLEU4 from 43.1 to 59.0 relative to OSCAR; on test it improves Micro-Acc from 76.5 to 78.5 and F1-BLEU4 from 41.9 to 53.0. Here the gap is framed as a failure to propagate global reasoning into follow-up generation (Zhang et al., 2022).

3. Commonsense inference and construct validity

The comprehension-generation gap is also visible when comprehension requires inference that shallow matching cannot supply. ReCoRD was created to bridge the gap between human and machine commonsense reading comprehension by filtering out examples solvable by a strong MRC model and emphasizing commonsense, multi-sentence, and partial-clue reasoning (Zhang et al., 2018). ReCoRD was built from 170K CNN/Daily Mail articles, yielding 770K raw triples, 244K after SAN filtering, and a final dataset with 100,730 train queries, 10,000 dev queries, and 10,000 test queries, across 80,121 unique passages. In a manual categorization of 100 development examples, 75% required commonsense reasoning, 10% were answerable via partial clues, 6% were ambiguous, and only 3% could be answered by straightforward paraphrasing. Human performance reaches 91.31 EM and 91.69 F1, while the best automatic system reported, DocQA with ELMo, reaches 45.44 EM and 46.65 F1. The gap here is not between two system modules, but between human inferential comprehension and machine performance under a benchmark designed to suppress lexical shortcuts (Zhang et al., 2018).

In diagnostic assessment, the gap becomes one of construct validity. "Automatic Generation of Inference Making Questions for Reading Comprehension Assessments" (Ma et al., 9 Jun 2025) introduces a taxonomy of inference types for reading comprehension: pronominal, pronominal bridging, text-connecting, and gap-filling. Using GPT-4o with few-shot prompting across four prompting conditions, the study evaluates 357 generated questions on general item quality, inference-type accuracy, and reasoning quality. The central result is a dissociation: 93.8% of generated items were judged good-quality overall, but only 42.6% matched the target inference type; in the best prompting condition, standard_6, inference-type accuracy reached 46.1%. Moreover, 34.8% of generated questions were factual or literal, and among target types, text-connecting was especially difficult at 24.1% match, compared with 60% for gap-filling and 53.3% for pronominal bridging. Chain-of-thought prompting did not clearly help. This is a precise instance in which fluent, answerable generation fails to preserve the intended inferential construct (Ma et al., 9 Jun 2025).

Taken together, these works separate three levels of difficulty: answer extraction, inferential comprehension, and faithful generation of a prompt or assessment item that targets that comprehension process. A plausible implication is that much of the empirical gap arises not because systems cannot generate language, but because they do not reliably preserve the underlying reasoning type that the task is meant to probe (Zhang et al., 2018, Ma et al., 9 Jun 2025).

4. Context, common ground, and interactive coupling

Several papers treat the gap as a failure to preserve context across interaction. "CoGen: Learning from Feedback with Coupled Comprehension and Generation" (Gul et al., 2024) studies two-player tangram reference games with a single multimodal model, IDEFICS2-8B, used as either speaker or listener. The paper argues that training or using the two capabilities in isolation creates drift: a better listener need not become a better speaker, and speaker-only continual learning can move generation away from human language. CoGen addresses this by coupling the two roles at training time through data sharing and at inference time through joint inference. In human deployment over four rounds, the FULL system improves from 53.31% to 72.79% in comprehension and from 52.00% to 78.07% in generation, and exceeds the non-coupled BASELINE by 14.80 points on comprehension and 17.10 points on generation at the end of the study. Qualitatively, FULL moves closer to human language under MAUVE and exhibits the highest Shape Naming Divergence throughout. The claim is that comprehension and generation become more effective when each role is informative for the other throughout learning and inference (Gul et al., 2024).

A social-conversational variant appears in "Bridging Context Gaps: Enhancing Comprehension in Long-Form Social Conversations Through Contextualized Excerpts" (Mohanty et al., 2024). Here the gap is the loss of background when a salient excerpt is detached from the conversation in which it originally occurred. The paper formalizes a conversation set $C = \{C_1, C_2, \dots, C_n\}$ with salient excerpts $E_j$, and seeks a context-enriched excerpt $\tilde{e}_{jl} = f(C_j, e_{jl})$. The Human-annotated Salient Excerpts dataset contains 1,774 salient excerpts across 177 conversations, and 90 excerpts were additionally annotated with social and contextual attributes. Two methods are compared: implicit contextualization $T_S$0 and explicit contextualization $T_S$1. Human ratings show statistically significant gains over the original excerpt; for example, understandability rises from 3.43 for the original excerpt to 4.11 for implicit contextualization and 4.13 for explicit contextualization, readability from 3.40 to 3.90 and 4.10, and empathy from 3.50 to 3.59 and 3.66. Explicit contextualization is shorter but more readable, coherent, and socially useful. At the same time, models remain weaker on long-response social factors such as background and personal experiences, and they clarified only 55 of 128 flagged extrinsic-knowledge instances in human evaluation. This is a context-reconstruction version of the comprehension-generation gap: faithful excerpt generation is insufficient when the social meaning needed for comprehension is omitted (Mohanty et al., 2024).

These studies make context a central variable. In one case, the relevant missing structure is pragmatic common ground in interaction; in the other, it is speaker identity, motivation, and setting. Both suggest that generation quality improves when systems can recover the latent state that originally made the utterance interpretable (Gul et al., 2024, Mohanty et al., 2024).

5. Unified multimodal models and bidirectional evaluation

In unified multimodal modeling, the comprehension-generation gap is often treated as an architectural problem. "SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation" (Ge et al., 2024) frames the gap as a mismatch between semantic comprehension and visual generation. Its solution is a shared visual tokenizer and de-tokenizer: a pretrained ViT maps images to visual embeddings, the MLLM predicts visual semantic representations with $T_S$2 learnable queries and MSE regression, and a de-tokenizer initialized from SDXL renders them as images. Dynamic-resolution encoding addresses arbitrary image sizes and ratios, while two-stage de-tokenizer training separates semantically aligned image generation from detail-preserving manipulation. TokLIP addresses the same problem differently by keeping low-level discrete VQ tokens for faithful autoregressive generation and adding high-level CLIP-like continuous semantics for comprehension, with contrastive and distillation objectives on top of a frozen VQGAN encoder (Lin et al., 8 May 2025). TokLIP uses less than 20% of the pretraining data required by VILA-U for strong representation learning, reaches 76.4% ImageNet Top-1 at 256 resolution, and improves ImageNet generation FID, for example from 14.48 to 12.37 for GPT-B at 384 resolution and 50 epochs (Lin et al., 8 May 2025). Divot extends the unification problem to video by learning a diffusion-powered continuous tokenizer whose features condition a diffusion decoder and also feed an LLM for comprehension; Divot-LLM reports 46.5 on EgoSchema, 58.3 on Perception-Test, 52.1 on MVBench, 76.4 on MSVD, and 55.8 on ActivityNet, while also achieving CLIPSIM 0.2938 and FVD 301.4 on MSR-VTT text-to-video generation (Ge et al., 2024).

Other work argues that architectural unification does not guarantee transfer from understanding to generation. "Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward" (Niu et al., 25 Nov 2025) introduces UniSandbox with synthetic reasoning-generation and knowledge-transfer tasks. Without explicit CoT, open-source unified models are near zero on reasoning-generation tasks; BAGEL averages 0.0283, Janus-Pro-7B and Blip3o about 0.0167, and Qwen-Image 0.0500. With explicit CoT, BAGEL + CoT reaches 0.5100 average, and in knowledge transfer BAGEL moves from 0.10 forward and 0.00 inverse to 0.63 forward and 0.15 inverse. The paper treats CoT as a bridge and proposes STARS, Self-Training with Rejection Sampling, to distill explicit reasoning into standard generation (Niu et al., 25 Nov 2025). GapEval provides a broader benchmark of 646 high-quality questions, each answerable in text and image, and models the gap with a multidimensional item-response formulation. The raw modality gap is

$T_S$3

with normalized absolute score

$T_S$4

Across a wide range of UMMs, the paper reports a persistent gap and argues that current systems achieve surface-level unification rather than deep cognitive convergence (Wang et al., 2 Feb 2026).

History-dependent interaction makes the gap more stringent. WEAVE defines the problem as comprehension-generation collaboration over dialogue history rather than single-turn comprehension or editing. WEAVE-100k contains 100,750 chats, over 370K dialogue turns, and 505,186 images; WEAVEBench has 100 tasks based on 480 images and 16 task categories. Training on WEAVE-100k improves a Bagel model by 9.8% on MMMU, 4.8% overall on GEditBench, 100% on both Spatial and Logical reasoning in RISEBench, and 42.5% overall on WEAVEBench, with 34.6% improvement on the harder science questions. Yet performance degrades as content length increases, open-source models often degrade when more context is provided, and multi-turn generation remains difficult even for the strongest systems (Chow et al., 14 Nov 2025).

The multimodal literature therefore distinguishes two claims. One claim is constructive: shared token spaces, semantic bottlenecks, and coupled training can reduce the gap. The other is diagnostic: even when understanding and generation coexist in one model, the two directions often remain unsynchronized, partially separate, or weakly transferable (Ge et al., 2024, Lin et al., 8 May 2025, Ge et al., 2024, Niu et al., 25 Nov 2025, Wang et al., 2 Feb 2026, Chow et al., 14 Nov 2025).

6. Software engineering manifestations

In software engineering, the gap is observed not between text and image, but between generated code or task completion and human understanding of the codebase. "Comprehension-Performance Gap in GenAI-Assisted Brownfield Programming: A Replication and Extension" (Qiao et al., 4 Nov 2025) studies 18 computer science graduate students completing brownfield feature implementation tasks with and without GitHub Copilot. The task used a 3,818-line front-end subset of a 100K-line sports analytics web application, and comprehension was measured with eight post-task questions worth 13 points total. Copilot reduced Task 1 completion time from 1311.6 seconds to 678.9 seconds, a 48.2% reduction, and increased passed tests from 3.9 to 7.2 on average, an 84% increase. Yet comprehension scores did not differ significantly: 7.9/13 without Copilot versus 7.2/13 with Copilot. Pearson correlations between total tests passed and comprehension score were 0.35 in the No Copilot condition and -0.25 in the Copilot condition, both nonsignificant. The study therefore defines a comprehension-performance gap: better apparent productivity without improved understanding of the legacy system (Qiao et al., 4 Nov 2025).

"Comprehension Debt in GenAI-Assisted Software Engineering Projects" (Ahmad, 14 Apr 2026) extends this concern into a socio-cognitive account of team knowledge. Based on 621 reflective diaries from 207 students over eight weeks, the paper defines Comprehension Debt as the growing gap between what a development team knows about its codebase and what it actually needs to understand in order to maintain and modify it effectively. Four accumulation patterns are identified: AI-as-black-box code acceptance, context-mismatch debt, dependency-induced atrophy, and verification-bypass. The mitigating pattern is GenAI as a comprehension scaffold. The paper further distinguishes acceleration orientation from exploration orientation: the former treats AI as a solution provider and reduces germane load, while the latter treats AI as a scaffold and supports deeper understanding. This recasts the comprehension-generation gap as a persistent imbalance between code-generation velocity and human comprehension capacity (Ahmad, 14 Apr 2026).

These studies are notable because the artifact may work. The failure appears later, when maintenance, debugging, or extension requires a mental model that was never built. In this domain, the gap is not merely an evaluation issue; it becomes a form of cognitive debt (Qiao et al., 4 Nov 2025, Ahmad, 14 Apr 2026).

7. Cross-cutting mechanisms and unresolved issues

Across these domains, one recurrent mechanism is missing context. INQUISITIVE shows that question generation improves when the model sees the current sentence and broader article context rather than only a highlighted span. CoGen improves performance by sharing positive interaction data across roles and by joint inference. Contextualized excerpts improve understandability and empathy by restoring speaker background, motivations, and situational details. WEAVE shows that multi-turn visual generation degrades as history length increases. This suggests that the gap frequently emerges when systems can produce outputs locally, but cannot reconstruct the latent discourse state, interaction history, or common ground required for those outputs to be appropriate (Ko et al., 2020, Gul et al., 2024, Mohanty et al., 2024, Chow et al., 14 Nov 2025).

A second recurrent mechanism is misalignment between surface success and target construct. In diagnostic reading assessment, a question can be fluent and operationally usable yet still miss the intended inference type. In unified multimodal models, a system can answer correctly in text while failing to generate the corresponding image, or vice versa. In brownfield programming, more tests can pass without any increase in comprehension. In Comprehension Debt, working code can accumulate while team understanding falls behind. The shared lesson is that output quality, benchmark success, or productivity are weaker criteria than faithful preservation of the relevant inferential structure (Ma et al., 9 Jun 2025, Wang et al., 2 Feb 2026, Qiao et al., 4 Nov 2025, Ahmad, 14 Apr 2026).

A third recurrent mechanism is weak transfer between modules or modalities. EFT addresses this by fusing entailment reasoning with generation in a one-stage model. SEED-X, TokLIP, and Divot attempt to create a shared semantic interface for both understanding and generation. UniSandbox shows that explicit CoT can bridge reasoning generation and knowledge transfer, while GapEval shows that even unified architectures still exhibit a persistent text-image gap. This suggests that closing the gap requires more than co-locating capabilities inside one model; it requires training and inference procedures that make intermediate reasoning, retrieved knowledge, or shared semantics usable by the generative pathway (Zhang et al., 2022, Ge et al., 2024, Lin et al., 8 May 2025, Ge et al., 2024, Niu et al., 25 Nov 2025, Wang et al., 2 Feb 2026).

The literature converges on several partial remedies rather than a single solution. These include sentence-level and discourse-level conditioning, common-ground filtering, hybrid generation-plus-human-review workflows for assessment, coupled listener-speaker learning, explicit reasoning traces, self-training with rejection sampling, curriculum learning, and tool designs that support a comprehension mode rather than only a generation mode. A plausible synthesis is that the comprehension-generation gap persists when systems optimize for local output adequacy, and narrows when they are forced to preserve the structure that made the output meaningful in the first place (Ma et al., 9 Jun 2025, Gul et al., 2024, Niu et al., 25 Nov 2025, Qiao et al., 4 Nov 2025).