Visual Query Pre-processing (V-QPP)
- Visual Query Pre-processing (V-QPP) is a method that refines raw visual or multimodal queries into structured forms to reduce ambiguity and better reflect user intent.
- It employs techniques such as geometric grouping, tool-based enhancement, and logical diagramming to extract and canonicalize semantic structures from noisy inputs.
- V-QPP improves system performance across applications like deep-web interfaces, knowledge-based visual Q&A, and SQL visualization by enhancing query accuracy and reliability.
Visual Query Pre-processing (V-QPP) denotes pre-retrieval or pre-execution transformations that convert an initial visual or multimodal query into a representation better aligned with downstream retrieval, reasoning, or execution. In the literature, the term spans several concrete settings: the conversion of visually designed deep-web query interfaces into structured, machine-interpretable query representations (Boughamoura et al., 2012), the extraction of canonical logical queries from visual interfaces through hierarchical grouping and schema alignment (Hlaoua et al., 2012), the construction of structured multimodal retrieval inputs for knowledge-based visual question answering (Hong et al., 16 Oct 2025), the agentic repair of imperfect visual queries before multimodal retrieval-augmented generation (Zhang et al., 13 Feb 2026), and the transformation of SQL into minimal, unambiguous logic-based diagrams that expose query intent before execution (Leventidis et al., 2020). Across these settings, the common objective is to reduce ambiguity, expose latent structure, and present downstream systems with a query representation that more faithfully reflects user intent.
1. Core concept and problem setting
In its contemporary retrieval-oriented form, V-QPP is defined as the active, agentic refinement of visual queries before retrieval in multimodal retrieval-augmented generation. The query is not treated as fixed input; instead, a model diagnoses imperfections, selects tools, predicts parameters, and emits a refined image for retrieval. In knowledge-based visual question answering, the same general idea appears as the principled transformation of raw multimodal inputs into a precise, structured multimodal query designed to maximize retrieval relevance and answer accuracy. In deep-web interface interpretation, V-QPP denotes the stage that converts a visually designed query interface into a structured query representation that preserves the interface’s semantic organization. In SQL-oriented work, the term is extended to a human-centered pre-processing layer that transforms parsed SQL into minimal, unambiguous diagrams before execution (Zhang et al., 13 Feb 2026, Hong et al., 16 Oct 2025, Boughamoura et al., 2012, Leventidis et al., 2020).
The problems addressed by these formulations differ, but their failure modes are closely related. In multimodal retrieval, real-world queries may contain rotation, flips, perspective skew, under- or over-exposure, blur, noise, aggressive cropping, overlays, watermarks, or cluttered scenes with multiple competing objects. In KB-VQA, full-image features may overemphasize salient but irrelevant content, miss small entities, or suffer from weak cross-modal alignment. In interface understanding, visually adjacent and aligned controls may not be recoverable from source-code order alone, so the visual layer becomes the primary signal for semantics. In SQL comprehension, syntactic variability and poor syntactic locality obscure logical intent, particularly in deeply nested correlated subqueries. V-QPP, in each case, is the attempt to correct these mismatches before the main retrieval or execution stage (Zhang et al., 13 Feb 2026, Hong et al., 16 Oct 2025, Hlaoua et al., 2012, Leventidis et al., 2020).
A common misconception is that V-QPP refers only to image enhancement. The literature is broader. Some systems emphasize perceptual repair, such as rotation correction or cropping. Others emphasize structural interpretation, such as extracting a motion tree from a web form or translating SQL to a Logic Tree and then to a diagram. This suggests that V-QPP is best understood as a family of pre-processing regimes whose unifying property is not the modality of the operation, but its placement before retrieval, answering, or execution and its role in improving semantic fidelity.
2. Interface-centered origins: visual interpretation and structured extraction
The earliest work in this corpus operationalizes V-QPP for deep-web query interfaces. VIQI models the input as a rendered web page with visible form controls, nearby textual labels, and bounding rectangles. Its output is a hierarchical query representation built from five element types: field, RenderedGroup, RenderedCollection, NotRenderedGroup, and VisualBox. The central assumption is that designers place semantically related controls close together and aligned, so visual cues recover the conceptual structure of the query. The basic geometric formulation is explicit: for fields and with bounding rectangles and ,
with binary alignment indicators for and an overall alignment score
A proximity-like dissimilarity is then defined as
0
and DBSCAN with adaptive 1 and typically 2 is used recursively to discover groups and super-groups until a top-level RenderedCollection is obtained (Boughamoura et al., 2012).
ViQIE presents a closely related formalization. It defines a visual query interface as a hierarchical “motion,” a tree of ordered items in which leaves are individual input fields and inner elements are semantically coherent groups of fields. To make the output machine-actionable and cross-service, it introduces a canonical interface schema
3
where 4 is the set of fields, 5 the associated labels, 6 inferred data types, 7 admissible domains, and 8 admissible operators. A pre-processed query is represented as
9
Cross-service composition is expressed through a mapping function
0
where 1 is a canonical schema of service-agnostic concepts such as DepartureCity, DestinationCity, DepartDate, ReturnDate, CabinClass, Adults, Children, and Infants. ViQIE also formalizes visual distance by combining Euclidean distance with alignment:
2
and uses DBSCAN recursively to recover GroupeAffiché, GroupeNonAffiché, and CollectionAffiché structures (Hlaoua et al., 2012).
The significance of these interface-centered systems is twofold. First, they treat visual layout as a semantic substrate rather than as presentation noise. Second, they frame query pre-processing as canonicalization: an interface-specific arrangement is transformed into a structured logical query that can be projected into multiple services while preserving service autonomy. Reported results indicate that precision decreases with query complexity, with flight or airfare forms consistently harder than book forms. The reported precision values include approximately 3 in Flights or Airfare and up to approximately 4 in Books on ICQ and TEL-8, reflecting the greater complexity of multi-constraint travel interfaces (Boughamoura et al., 2012, Hlaoua et al., 2012).
3. Structured multimodal query construction in KB-VQA
In knowledge-based visual question answering, Wiki-PRF instantiates V-QPP as the Processing stage of a three-stage pipeline: Processing, Retrieval, and Filtering. The Processing stage dynamically invokes visual tools to produce high-quality multimodal retrieval inputs. The tool set consists of captioning, grounding, and flipping. Captioning generates a question-conditioned description 5; grounding identifies and crops the core subject to produce 6; flipping returns 7 for orientation normalization. The output of pre-processing is a structured multimodal query,
8
which is then passed to multimodal retrieval. Orchestration is handled by a reasoning VLM, VLM-PRF, which writes plans inside > ..., emits executable tool plans in <tool>...</tool>, and uses Qwen2.5-VL as the execution substrate (Hong et al., 16 Oct 2025).
The retrieval formulation combines image- and text-based evidence using EVA-CLIP-8B embeddings and Faiss-GPU cosine similarity. For tool image-level retrieval, if 9 and 0, the paper defines
1
yielding tool-driven Top-2 article or image candidates. For section selection, with articles split into sections 3 and 4,
5
The final retrieved sections 6 are the union over tool calls, while 7 is the Top-1 article retrieved directly from the image. The Filtering stage then consumes 8 and 9 and emits compact task-oriented knowledge 0 within <answer> (Hong et al., 16 Oct 2025).
Wiki-PRF further couples V-QPP to reinforcement learning. VLM-PRF is trained with answer accuracy and format compliance rewards. The general objective is
1
with
2
The paper specifies a GRPO-style clipped surrogate and a reward
3
with reported values 4 in the main setting and 5 in later experiments. Empirically, Wiki-PRF with RL and Qwen2.5-VL-7B reaches 6 on E-VQA All and 7 on InfoSeek All, while the no-RL version reaches 8 and 9, respectively. On a 0 InfoSeek subset with Qwen2.5-VL-3B, retrieval only scores 1, adding Filtering yields 2, adding Processing yields 3, and combining Processing and Filtering yields 4. Tool ablations report 5 for grounding only, 6 for captioning only, and 7 for multi-tools. Retrieval recall on InfoSeek also improves: direct image Top-1 is 8, images plus tools post-RL reaches 9 for 0B and 1 for 2B (Hong et al., 16 Oct 2025).
4. Agentic correction before multimodal retrieval
“Fix Before Search” recasts V-QPP as an explicit agentic decision-making problem for MRAG. The observed query image is modeled as a composition of corruptions,
3
which induces embedding misalignment,
4
The agent has a toolbox 5, predicts a sequence 6, and composes tools to produce
7
Retrieval is then optimized through
8
The paper also gives a supervised objective
9
while the implemented SFT uses sequence-to-JSON token cross-entropy (Zhang et al., 13 Feb 2026).
The benchmark, V-QPP-Bench, contains 0 imperfect queries built from 1 clean images, each corrupted by ten imperfection types with randomized parameters. The imperfection taxonomy includes geometric distortions, quality degradations, and semantic ambiguity: Rotation 2, Flip 3, Brightness 4, Blur 5, Noise 6, Crop 7, Expand, Overlay, Watermark, and RealWorld. The tool library includes OpenCV rotation, flip, and brightness adjustment; Richardson–Lucy deblurring with a 8 PSF and 9 iterations; non-local means denoising; Grounding DINO for localization; deterministic crop; and fill for masking overlays or watermarks. Retrieval paradigms include Dense I2T, Caption-then-retrieve, I2I, I+T→I+T, and Search APIs such as Google Lens (Zhang et al., 13 Feb 2026).
The quantitative findings are unusually sharp. Average Recall@5 drops in I2T relative to clean queries by 0 for RealWorld, 1 for Expand, 2 for Rotation, 3 for Flip, 4 for Overlay, 5 for Crop, 6 for Noise, 7 for Blur, and 8 for Brightness; Watermark has a small impact on recall, approximately 9, but can mislead generation. In Google Lens experiments on a 0-sample subset, end-to-end accuracy falls from 1 on original inputs to an average 2 under imperfections, while oracle preprocessing improves the average to 3. Off-the-shelf MLLMs exhibit an “agentic bottleneck” in tool selection and parameter prediction, but supervised fine-tuning on only 4 training instances enables Qwen3-VL-4B SFT to recover performance broadly and reach 5 on Google Lens, compared with 6 for the zero-shot agent. The paper concludes that compact agents can match or beat much larger proprietary models on V-QPP when trained specifically for tool use and parameter estimation (Zhang et al., 13 Feb 2026).
5. Logic-based visual pre-processing for SQL comprehension
QueryVis extends the notion of V-QPP beyond image retrieval to pre-execution query understanding. It translates SQL into first-order-logic-based diagrams designed to be minimal and unambiguous. The supported fragment includes nested conjunctive queries with inequalities, WHERE subqueries with EXISTS/NOT EXISTS, IN/NOT IN, ANY, and ALL, with join operators in 7. The pipeline is: parse SQL, convert it to Tuple Relational Calculus, build a Logic Tree whose nodes are query blocks annotated with tables, predicates, and quantifier, optionally apply De Morgan transformations to simplify nested NOT EXISTS into universal quantification, and render the result as a diagram with table composite marks, dashed rounded rectangles for NOT EXISTS, double-lined rounded rectangles for FOR ALL, and edges for join predicates. The paper’s formal statement of minimality is “DEFINITION 4.1,” and its unambiguity result is “PROPOSITION 5.1 (UNAMBIGUITY): For any valid QUERYVIS diagram there exists exactly one LT that maps to it” (Leventidis et al., 2020).
The logical normalization step is central. The paper gives the transformation
8
to
9
making quantifier scope explicit in the visualization. Reading order is defined as a depth-first traversal from the SELECT table with restarts from unvisited source nodes. This means that the diagrams are intended not merely as pictures of syntax, but as normalized representations of logical intent. The relation to V-QPP is therefore interpretive rather than perceptual: the query is made legible before execution, review, refactoring, or reuse (Leventidis et al., 2020).
The empirical evaluation involved 00 legitimate AMT workers, with 01 illegitimate participants excluded. On nine multiple-choice questions over the Chinook schema, QueryVis reduced median completion time by 02 relative to SQL alone, from 03 to 04, with 05. Mean error rate declined from 06 to 07, a 08 reduction, although with 09. These results support the claim that visual pre-processing can improve comprehension even for expert-oriented formal query languages, provided the representation is minimal, logically faithful, and unambiguous (Leventidis et al., 2020).
6. Limitations, misconceptions, and open directions
The literature identifies several recurring limitations. In visual interface parsing, highly dynamic interfaces, hidden or conditional fields, responsive designs, non-textual labels, and unusual layouts weaken the assumptions behind proximity and alignment; parameter sensitivity, especially the choice of 10 in DBSCAN, can fragment or over-merge groups (Boughamoura et al., 2012, Hlaoua et al., 2012). In KB-VQA, mis-grounding, orientation effects, and weak cross-modal alignment can still impair retrieval, and the paper notes that captioning, grounding, and flipping are the tools actually implemented, while OCR and entity linking are discussed more as general V-QPP principles than as components of the specific system (Hong et al., 16 Oct 2025). In MRAG, irreversible corruptions such as heavy Crop or severe Blur/Noise cap restoration performance, and parameter uncertainty in angles or bounding boxes propagates directly to retrieval failure; the paper explicitly recommends conservative sequencing, confidence-triggered invocation, and fallbacks to the raw path when refinement lowers retrieval scores (Zhang et al., 13 Feb 2026). In QueryVis, the formal core excludes disjunctions, NULL semantics, and aggregates during the FOL transformation step, with GROUP BY treated only as an exploratory extension (Leventidis et al., 2020).
Several misconceptions are directly contradicted by the reported results. One is that visual query pre-processing contributes only marginally to system behavior. The benchmarked MRAG setting shows the opposite: visual imperfections can reduce Recall@5 by as much as 11 in RealWorld and 12 in Expand, while oracle preprocessing nearly restores performance for reversible distortions (Zhang et al., 13 Feb 2026). Another is that larger general-purpose MLLMs are sufficient for pre-processing without specialized training. The supervised fine-tuning results on Qwen3-VL-4B show that a compact model trained on 13 oracle traces can surpass larger off-the-shelf systems on tool selection and parameter prediction (Zhang et al., 13 Feb 2026). A third is that V-QPP is inherently tied to image repair. The broader record shows deterministic geometric grouping for forms, tool-driven multimodal query construction for KB-VQA, agentic query repair for MRAG, and logic-based SQL visualization for human interpretation (Boughamoura et al., 2012, Hong et al., 16 Oct 2025, Leventidis et al., 2020).
Open directions are explicit in the sources. The MRAG benchmark proposes extending the tool set with dewarping, perspective normalization, super-resolution, OCR-guided enhancement, adaptive multi-step sequencing, co-training of retriever and preprocessor, and reinforcement learning for tool use (Zhang et al., 13 Feb 2026). Wiki-PRF suggests applicability beyond KB-VQA to Chart QA, Document VQA, and Scene QA, with additional OCR, layout parsing, or spatial relation extraction depending on the domain (Hong et al., 16 Oct 2025). The interface interpretation literature points toward learning-based label-field association, adaptive clustering thresholds, headless-browser support for dynamic forms, and broader ontologies for cross-service alignment (Hlaoua et al., 2012). QueryVis identifies extensions to aggregation, HAVING, arithmetic predicates, NULLs, broader nesting, and interactive IDE integration (Leventidis et al., 2020). Taken together, these directions indicate that V-QPP is moving from handcrafted pre-processing heuristics toward explicit, trainable, and diagnosable front-end modules that mediate between noisy visual input and structured downstream reasoning.