Cross Modal Pre Questions

• Cross Modal Pre Questions (preQs) are automatically generated, task-agnostic queries that capture rich multimodal context from text, images, and tables.
• They employ a multi-stage pipeline involving document parsing, multimodal query construction, embedding, and clustering to sharpen retrieval precision.
• Empirical evaluations show that preQs significantly boost Recall and MRR metrics over traditional monolithic vector methods across diverse benchmarks.

Cross modal pre questions (preQs) are cross-modality, task-agnostic queries generated from multimodal content—such as documents or datasets containing text, images, tables, or speech—with the explicit design intent of enhancing downstream retrieval, reasoning, or question-answering tasks. They operate at the intersection of question generation, multimodal representation learning, and LLM inference, and are most recently formalized and empirically evaluated by their role in token-level multimodal retrieval systems such as the PREMIR framework (Choi et al., 23 Aug 2025).

1. Foundations of Cross Modal Pre Questions

Cross modal pre questions (preQs) are not traditional queries posed by users but are instead context-sensitive questions automatically generated by an LLM or a similarly capable MLLM, conditioned on the parsed multimodal content of a real-world document. The distinctive property of preQs is their cross-modality: they can be instantiated using information from images, text extracted via OCR, tables, figures, or any combination thereof. Generation is performed prior to retrieval, so preQs act as fine-grained, semantically-rich “tokens” or proxies for conceptual content distributed across a multimodal information space.

Unlike previous retrieval methods that embed entire documents or pages as monolithic vectors, the preQ paradigm decomposes content into a diverse set of representation anchors, each capturing explicit or implicit knowledge in a modality-aware fashion. This enables token-level, modality-complementary matching and expands the retrieval capacity well beyond the limitations of single-vector representations (Choi et al., 23 Aug 2025).

2. PreQ Generation and Methodological Architecture

The generation and utilization of cross modal pre questions usually follows a multi-stage, pipeline methodology:

1. Document Parsing and Multimodal Decomposition: Incoming documents are decomposed by a layout-aware parser, with raw page images, OCR text, and separate identified components (e.g., tables, figures, images) extracted. Each visual component is also captioned using an MLLM, and all textual elements (including captions) are merged to create a layout-preserving "textual surrogate" that mirrors human reading order.
2. Multimodal PreQ Construction:
   • Multimodal preQs (Pₘ): Generated by prompting the LLM with the entire raw page image, yielding queries that capture holistic document context.
   • Visual preQs (Pᵥ): Derived from LLM interpretation of individual visual elements, targeting figure-specific or table-centric reasoning.
   • Textual preQs (Pₜ): Created from the textual surrogate, emphasizing granular linguistic details.

   The overall candidate preQ pool for each page is then defined by:

   $P = Pₘ \cup Pᵥ \cup Pₜ$

3. Embedding and Pooling: All generated preQs for each page are embedded using a retriever backbone; the user's query (whether text or multimodal) is similarly embedded.
4. Q-Cluster Retrieval: Top-k preQs are retrieved by semantic similarity (often via cosine distance), then grouped (clustered) according to their source page/component. An LLM-based scoring mechanism ranks these clusters, promoting those with the highest aggregate relevance to the original query.

This process enables the retriever to go beyond monolithic matching, leveraging the semantic diversity and multigranularity of the preQ set.

3. Empirical Performance and Ablation

Empirical studies in PREMIR and related frameworks demonstrate several key findings (Choi et al., 23 Aug 2025):

• Superior Retrieval Metrics: On challenging out-of-distribution (OOD), closed-domain, and multilingual retrieval benchmarks (e.g., ViDoSeek, REAL-MM-RAG, CT²C-QA, Allganize Korean), preQ-powered retrieval achieves state-of-the-art Recall@1, Recall@3, Recall@5, and MRR@5, consistently outperforming both text-chunk and image-chunk embedding baselines.
• Ablation Insights: Removing any single preQ modality (multimodal, visual, or textual) results in performance degradation, but the greatest metric drops occur when the clustering (Q-Cluster) step is omitted—evidence that aggregating preQs at the document context level is essential for filtering generic or spurious matches.
• Robust to Embedding Model Choice: Performance gains from preQs are consistent even when the embedding model is changed from proprietary to open-weight variants (BGE, Qwen2-based GTE), indicating that it is the preQ representation structure—rather than just embedding model power—that underpins the observed improvements.

A representative performance summary is shown below:

Benchmark	Recall@1 (PreMIR)	SOTA Baseline
ViDoSeek (Closed)	0.797	0.627 (ColQwen2.0)
CT²C-QA (Open)	0.255	0.126 (ColQwen2.0)
Allganize KR	0.760	0.661 (ColBERT)

4. Qualitative Properties and Analysis

PreQs generated from different modalities offer complementary perspectives:

• Multimodal preQs encapsulate document-wide narratives, useful for context-heavy searches.
• Visual preQs yield focused queries about elements like plots, images, or structural figures; they are critical for visual-related question answering and for retrieval when visual salience is crucial.
• Textual preQs capture specific linguistic details, dissect definitions, and enable precise semantic anchoring for queries about entities and relationships.

Qualitative visualization (e.g., t-SNE projections) underscores that retrieved top preQs cluster well according to user intent, showing clear separation between high-salience and distractor documents, and that Q-Cluster reranking demotes generic or weakly relevant preQs.

5. Engineering and Modeling Implications

The preQ paradigm brings a series of methodological shifts and influences:

• Token-Level, Modality-Specific Granularity: Retrieval transitions from holistic page-matching to fine-grained, contextually-resolved matching, significantly enhancing precision.
• Flexible to Domain and Modality: The approach readily extends to documents rich in figures, images, or non-textual elements; preQs can be generated and matched in multilingual and cross-domain settings with competitive robustness.
• Synergy with Modern MLLMs: As MLLMs such as GPT-4o build broad prior knowledge across domains and modalities, the quality and variety of preQs improve correspondingly, yielding better coverage for OOD or low-resource domains.

6. Limitations, Challenges, and Future Directions

While cross modal pre questions have established superior retrieval and reasoning performance, several open challenges persist (Choi et al., 23 Aug 2025):

• Quality Control: Not all LLM-generated preQs are specific or non-trivial; managing the generation process to minimize generic or repetitious questions may require prompt engineering or post-processing filters.
• Computational Complexity: The token-level representation strategy can scale computational cost if the number of preQs is not carefully managed, which suggests the need for adaptive preQ number selection techniques based on document complexity.
• Clustering and Ranking: The effectiveness of Q-Cluster aggregation may vary with the underlying semantic heterogeneity of preQs. Further research may focus on alternative clustering algorithms and the integration of external domain knowledge for reranking.

Future Work: Promising directions include refining preQ specificity, dynamic preQ pool sizing, adaptation to new document types (e.g., diagrams, spatial layouts), and integrating self-improving feedback loops where downstream retrieval performance guides preQ generation parameters.

7. Context and Significance in Multimodal AI

Cross modal pre questions represent a substantial advance in multimodal information retrieval and understanding, particularly within the context of retrieval systems, robust multi-domain question answering, and cross-lingual adaptation. By leveraging the semantic decomposition afforded by both content parsing and MLLM-powered question generation, the preQ framework transforms retrieval from a static vector-based paradigm to an active, query-centered process capable of leveraging the full breadth of multimodal, multilingual document content (Choi et al., 23 Aug 2025).

This approach establishes a precedent for future research at the intersection of question generation, cross-modal fusion, and representation learning—pointing toward systems that can both interpret and proactively interrogate complex, realistic information environments.