WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (2511.11434v1)

Abstract: Recent advances in unified multimodal models (UMMs) have enabled impressive progress in visual comprehension and generation. However, existing datasets and benchmarks focus primarily on single-turn interactions, failing to capture the multi-turn, context-dependent nature of real-world image creation and editing. To address this gap, we present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation. Our suite consists of two complementary parts. WEAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images, covering comprehension, editing, and generation tasks that require reasoning over historical context. WEAVEBench is a human-annotated benchmark with 100 tasks based on 480 images, featuring a hybrid VLM judger evaluation framework based on both the reference image and the combination of the original image with editing instructions that assesses models' abilities in multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments demonstrate that training on WEAVE-100k enables vision comprehension, image editing, and comprehension-generation collaboration capabilities. Furthermore, it facilitates UMMs to develop emergent visual-memory capabilities, while extensive evaluations on WEAVEBench expose the persistent limitations and challenges of current approaches in multi-turn, context-aware image generation and editing. We believe WEAVE provides a view and foundation for studying in-context interleaved comprehension and generation for multi-modal community.

• The paper presents WEAVE, a dataset and benchmark suite for multi-turn, context-aware multimodal comprehension, editing, and generation.
• It employs a sequential annotation pipeline with four rigorous methods to integrate textual and visual interactions with historical context.
• Experimental results show that even leading UMMs struggle with extended context, while WEAVE fine-tuning enhances visual memory and reasoning.

WEAVE: Benchmarking In-context Interleaved Multimodal Comprehension and Generation

Motivation and Distinctiveness

Recent advancements in Unified Multimodal Models (UMMs) have delivered formidable visual comprehension and generation capabilities. Nevertheless, the majority of existing datasets and evaluation protocols narrowly emphasize single-turn interactions, failing to capture the iterative, context-dependent workflows intrinsic to authentic image creation and modification. The authors introduce weaveW, posited as the first large-scale benchmark suite designed for multi-turn, context-aware comprehension, generation, and editing in a multimodal setting.

Figure 1: Existing datasets focus on simple single-turn overlays, whereas weaveW supports multi-turn edits requiring visual memory recall.

WEAVE’s core novelty derives from interleaving textual and visual modalities over extended dialog horizons, systematically encoding temporal dependencies and historical context. This is in contrast to previous works—where edits are isolated and lack long-horizon semantic or visual consistency.

Dataset Construction and Annotation Pipeline

WEAVE’s dataset component is composed of $100$K interleaved samples, totaling over $370$K dialogue turns and $500$K images, covering diverse tasks such as comprehension, editing, and generation demanding historical context reasoning. The annotation pipeline is engineered to maximize data diversity, quality, and memory-based reasoning. It utilizes four rigorous pathways:

1. Multi-image fusion: Edited or generated images are directly fused, requiring the model to reference results from previous rounds.
2. Remove-then-back: Objects are systematically removed and subsequently reintroduced, enforcing recall of previously deleted visual entities.
3. Derivative imagination/comparison: Models are tasked to derive or imagine alternative visual solutions before fusion, expanding the generative reasoning landscape.
4. Sequential procedures: Narratives, story arcs, or structured editing operations demand explicit inter-step relationships and visual consistency.

   Figure 2: WEAVE’s multi-stage annotation pipeline with iterative image generation, double-round validation, and refinement ensures visual memory-rich data streams.

Human annotation, especially by domain specialists, was adopted for the benchmark component, ensuring rich coverage in science, logic, narrative construction, games, and world knowledge.

Benchmark and Evaluation Protocols

The WEAVE benchmark spans 100 tasks leveraging 480 images. It diverges by introducing a hybrid Vision-LLM (VLM) judge evaluation framework that aggregates scoring based on:

• Key Point Correctness (KP): Satisfaction of explicit editing or generation requirements.
• Visual Consistency (VC): Preservation of non-targeted content and stylistic coherence.
• Image Quality (IQ): Assessment of generative fidelity.
• Accuracy (Acc): Evaluates correctness for comprehension-centric tasks.

Figure 3: Quantitative ablation reveals the criticality of in-context input modes, sequential image arrangement, and the evaluation reliability of GPT4.1 as a VLM judge.

Reference images and prompts are systematically constructed, leveraging ground-truth alignment and history tracking. Benchmark coverage spans multiple domains (Figure 4–25), evaluating both general and specialized reasoning.

Experimental Results and Analysis

The authors comprehensively evaluate 22 models (LLMs, editing models, and UMMs) under three in-context regimes: no history, partial history (explicit recent context), and complete history with all prior interactions available. Models are further tested using sequential and concatenated image input modes.

Key empirical findings include:

• Performance Ceiling: Even the best-performing UMMs and editing models reach only $0.767$ and $0.68$ in-context scores, respectively. The gap persists as dialog length increases, underscoring the unresolved challenge of context-dependent multimodal reasoning in current architectures.
• Domain Bias: Substantial bias favours creative generation over scientific and logical domains. For instance, QwenVL showed marked improvement with in-context information ($163\%$ boost in comprehension tasks).
• Sequential Superiority: Models utilizing sequential (not concatenated) image input outperform by up to $10.3\%$ (Bagel), emphasizing the importance of explicit temporal ordering in multi-turn editing tasks.
• Cross-modal Enhancement via WEAVE Training: Fine-tuning Bagel on weaveW led to a $4.8\%$ gain on GEditBench and $9.8\%$ on MMMU understanding, along with substantial gains (up to $100\%$) in RISE cognitive tasks for spatial and logical reasoning.

  Figure 5: Partial schematic overview of WEAVE’s multimodal interaction flows, exemplifying complex, multi-image dialogs.

  

  Figure 6: WEAVE statistics: majority of samples have >5 images and display a balanced multi-domain distribution.

  

  Figure 4: Example of astronomy domain testing the model's understanding of celestial objects and phenomena.

  

  Figure 7: Example from the chemistry domain, stressing the necessity of modality-specific world knowledge for accurate generation.

Fine-tuning and ablation studies isolate the emergent visual memory property and contextual adaptability conferred by WEAVE’s interleaved format.

Quality, Reliability, and Limitations

Quality analysis via human raters and VLM-as-judge (GPT-4.1) confirms evaluation reliability, with Pearson correlations exceeding $0.8$ for human-VLM agreement. Qualitative failures are apparent in instruction-following and nuanced memory recall. Fine-tuned models on WEAVE outperform on retention and recalling character identity, demonstrating the protocol’s capacity for benchmarking emergent properties.

Figure 8: Diverse image style references used for Nano Banana inference, critical for enhancing editing quality.

(Figures 11–14, 15–19)

Figures 11–19: Examples of multi-image fusion, sequential edits, and recall tasks; collectively probing models’ visual memory and narrative editing skill.

Practical and Theoretical Implications

WEAVE establishes a robust foundation for evaluating and analyzing context-aware multimodal reasoning in UMMs. Practically, its rigorous dataset and benchmark suite provide critical resources for systematic model training, evaluation, and comparison under realistic multi-turn conditions. The detailed annotation and evaluation pipeline facilitate straightforward reproducibility in open-source and commercial environments.

Theoretically, the persistent degradation of model accuracy with increased context length reveals limits in transformer-based architectures and context window utilization. The emergence of visual memory through interleaved training signals new directions for explicit memory module design and hierarchical context modeling.

Outlook and Potential Developments

Future research will likely investigate scaling strategies to enable token-efficient multi-turn dialog modeling, advanced visual memory architectures, and enhanced fusion of symbolic and visual modalities. The explicit evaluation protocols of WEAVE may catalyze more standardized benchmarks, driving progress at the intersection of editing, generation, and historic context reasoning in multimodal systems. There is also scope for domain specialization, focusing on scientific, logical, and narrative tasks to reduce the observed generation bias and address multimodal world knowledge integration.

Conclusion

WEAVE represents a substantive advancement in the evaluation and benchmarking of in-context, multi-turn multimodal comprehension and generation. Through its large-scale, richly annotated dataset and hybrid evaluation protocols, it exposes substantial limitations in current UMMs and suggests clear directions for architectural and training improvements. The emergence of visual memory and improved interleaved reasoning from WEAVE-trained models marks a notable shift towards more context-coherent and historically aware generative AI systems.