---
title: 'NL-DIR: Natural Language Document Image Retrieval'
url: https://www.emergentmind.com/topics/natural-language-based-document-image-retrieval-nl-dir
type: topic
---

# NL-DIR: Natural Language Document Image Retrieval

Natural Language-based Document Image Retrieval (NL-DIR) is a paradigm in document retrieval that targets the retrieval of document images from a corpus using semantically rich natural language queries. Unlike classic DIR approaches—often relying on image queries and retrieving only within coarse semantic categories—NL-DIR operates on real-world queries that express fine-grained details, intent, and visual elements, thereby facilitating retrieval scenarios better aligned with user needs in contemporary information management systems [2512.20174, 2505.11651].

## 1. Formal Problem Definition

NL-DIR is defined over a corpus $\mathcal{D} = \{d_1, \ldots, d_N\}$ of document images and a query set $\mathcal{Q} = \{q_1, \ldots, q_M\}$ comprised of natural-language prompts. Two embedding functions are learned:
$$
f_d: \mathcal{D} \to \mathbb{R}^D, \qquad f_q: \mathcal{Q} \to \mathbb{R}^D
$$
These map both images and queries into a shared $D$-dimensional latent space. Retrieval for a query $q$ consists of finding:
$$
d^* = \arg\max_{d \in \mathcal{D}} \mathrm{sim}(f_q(q), f_d(d))
$$
where $\mathrm{sim}(\cdot, \cdot)$ is typically dot-product or cosine similarity [2512.20174]. In multi-lingual settings, each query $q \in Q$ has a ground-truth page $d^*_q \in D$ such that:
$$
\text{top}_k(q) = \text{argmax}_{j \in D}^{k} s(q, d_j)
$$
with $s: Q \times D \to \mathbb{R}$ a scoring function; the retrieval task seeks $d^*_q \in \text{top}_k(q)$ [2505.11651].

## 2. Datasets and Annotation Protocols

### NL-DIR Dataset [2512.20174]
- **Composition:** 41,795 authentic scanned document images, each paired with five distinctive, fine-grained semantic queries, yielding $\sim$205,000 queries.
- **Categories:** 247 classes, e.g., letters, forms, reports, advertisements, charts (with 15 largest categories detailed).
- **Splits:** Train/val/test (8:1:1), with 4,180 test images serving as the benchmark.
- **Query Generation:** Automated OCR extraction with layout cues, LLM-driven query synthesis (ChatGPT produces 10 candidates/image), scored by ChatGPT, Qwen-VL-Plus, CLIP, and BLIP. Weighted aggregation (3:3:2:2), followed by manual inspection and curation.
- **Semantic Richness:** High query-OCR text overlap (≥3 words), but queries generalize to encompass layout, visuals, and intent beyond raw text.

### MIRACL-VISION Benchmark [2505.11651]
- **Corpus:** Multilingual, spanning 18 languages (English, Chinese, Arabic, Hindi, etc.), $\sim$7,900 queries, and $\sim$18,819 document page images per language.
- **Source:** First-page screenshots of Wikipedia articles; queries are human-generated (native speaker annotation).
- **Negative Sampling:** “Easy” negatives excluded via dense multilingual text embedding ranks, maintaining retrieval challenge.
- **Annotation Process:** Ensures one ground-truth page per query, with diverse queries not limited to paraphrases.

## 3. Retrieval Pipeline Architectures

### Baseline Modalities
- **Contrastive Vision-Language Models (VLMs) [2512.20174]:** Two-tower architectures (CLIP, BLIP, DFN, SigLIP, InternVL-14B):
  - Pretrained with InfoNCE or Sigmoid contrastive loss, mainly on natural images and some web/OCR data.
- **OCR-Free Visual Document Understanding (VDU) Models:** (Donut, Nougat, Pix2Struct, DocOwl1.5, UReader, TextMonkey, Qwen2-VL, DSE, ColPali)
  - Generative decoder-only models aimed at structured output or text extraction without explicit contrastive alignment.
  - Adapted by mean-pooled embeddings and dot-product scoring for zero-shot retrieval.

### Multilingual Pipeline (MIRACL-VISION) [2505.11651]
- **Text-only:** Dense text encoders (multilingual-e5-large, bge-m3, gte-multilingual-base, Llama-3 tuned for embeddings).
- **Visual LLM (VLM):** Vision encoder (e.g., SigLIP) for page images; text encoder for query; projection MLP for shared embedding alignment; contrastive pair training.
- **Precomputed Embeddings:** Use FAISS for fast top-$k$ candidate selection via cosine similarity.

## 4. Two-Stage Retrieval and Re-Ranking

### Coarse Recall [2512.20174]
- Precompute $f_d(d)$ for all gallery images (FAISS index, $\sim$0.07$\,$s/image, $\sim$4$\,$KB/image).
- For query $q$, encode $f_q(q)$, retrieve top-100 candidates by dot-product similarity.

### Fine Re-Ranking
- For top-100 candidates, apply cross-modal interaction models for refined scoring $s_\mathrm{cross}(q, d)$.
- Models: BLIP-ITM, Pix2Struct+ITM with additional cross-attention, trained with hard-negative mining (top-10 hard negatives; mix pointwise and pairwise loss).
- Reranking $\sim$0.2$\,$s/query for 100 images (parallel processing).

**Retrieval pseudocode:**
```python
for q in test_queries:
    C = recall_top100(q)
    for d in C:
        s[d] = CrossAttentionITM(f_q(q), f_img(d))
    results = sort_by_score(C, s)
```

## 5. Evaluation Metrics and Experimental Comparisons

### Metrics
- **Recall@$K$:** $\frac{1}{|\mathcal{Q}|} \sum_q \mathbf{1}[GT_d(q) \in \text{top-K}(q)]$
- **MRR@$K$:** $\frac{1}{|\mathcal{Q}|} \sum_q \frac{1}{\text{rank}_q}$ (truncated at $K$)
- **mAP** and **NDCG@$K$** (optional; standard IR metrics) [2512.20174, 2505.11651].

### Results on NL-DIR [2512.20174]
- **Zero-shot:** SigLIP-So400m $R@1$=36.17, $R@10$=61.18, MRR@10=43.78. InternVL-14B yields lower recall. VDU models have near-zero recall.
- **Fine-tuning:** SigLIP-Text-LoRA achieves $R@1$=54.3, $R@10$=79.4; SigLIP-Image-LoRA achieves $R@1$=69.3, $R@10$=89.7. Best $R@100$=97.52.
- **Re-ranking:** ColPali (zero-shot) $R@1$=79.6, $R@10$=91.6, MRR=83.8. Fine-tuned BLIP-ITM matches ColPali with fourfold lower storage.
- **Subset Analysis:** Concrete queries yield higher recall ($R@1$=86.2, $R@10$=97.2, $R@10$=99.4 after rerank); abstract queries are less effectively retrieved.

### Multilingual Gap in MIRACL-VISION [2505.11651]
- Best text embedding: NDCG@10=0.7964.
- Best VLM: NDCG@10=0.5283 (42% lower). On English, text vs. vision gap is 12.1%; in Hindi/Arabic up to 59.7%.
- VLM performance degrades sharply in non-Latin scripts and resource-poor languages.

| Language | Text NDCG@10 | Vision NDCG@10 | Drop (%) |
|----------|-------------|----------------|----------|
| Arabic   | 0.8883      | 0.4888         | 45.0     |
| Hindi    | 0.7581      | 0.3127         | 58.8     |
| English  | 0.7348      | 0.6784         | 7.7      |
| Telugu   | 0.9090      | 0.0893         | 90.2     |

## 6. Efficiency and Practical Considerations

- **Offline encoding:** SigLIP $\sim$0.07$\,$s/image, OCR-IR (Tesseract+BGE) $\sim$2.56$\,$s, DSE $\sim$0.62$\,$s, ColPali $\sim$0.65$\,$s per image [2512.20174].
- **Storage:** SigLIP 4 KB/image; BGE 3 KB; DSE 6 KB; ColPali 256 KB.
- **Online query/rerank:** Dot-product selection is sub-ms; reranking 0.2 s for 100 images.
- MIRACL-VISION employs aggressive hard-negative mining to render evaluation manageable yet challenging [2505.11651].

## 7. Challenges, Insights, and Future Directions

### Key Technical Obstacles
- Fine-grained semantics require models robust to domain shift from natural images to complex layouts, error-prone OCR, and visually rich content.
- Multilingual VLMs lag text-only models, with retrieval efficiency and semantic alignment suffering in non-Latin scripts due to training data gaps and OCR limitations.

### Observations
- Contrastive VLMs pretrained on image–OCR pairs (SigLIP) excel in zero-shot setting; cross-attention layers in reranking crucially improve fine-grained matching [2512.20174].
- OCR-free VDU models outperform when visual information dominates (e.g., charts, handwriting, cartoons).

### Recommendations [2505.11651]
- Build benchmarks from human-annotated queries, ensuring linguistic and script diversity; couple image and text versions for cross-modal gap analysis.
- Incorporate hard-negative mining to maintain retrieval challenge and tractability.
- Improve VLM accuracy via domain-adaptive multilingual fine-tuning, multilingual page image augmentation, OCR-aware vision encoder training, and hybrid fusion of visual/text embeddings.

### Future Research Directions [2512.20174]
- Develop specialized, lightweight cross-modal alignment architectures tuned for document genres.
- Integrate generative VDU and cross-attention in unified rerankers.
- Extend to multi-page retrieval and paragraph-level localization.
- Scale datasets and pre-training on document-centric corpora to bridge existing domain gaps.
- *This suggests* that addressing semantic and script diversity, while reducing reliance on synthetic LLM-generated queries, is essential for robust, real-world NL-DIR deployment.

## Summary

Natural Language-based Document Image Retrieval represents a significant advancement for document access via expressive natural language, characterized by challenging datasets, precise evaluation protocols, and sophisticated cross-modal architectures. The NL-DIR and MIRACL-VISION benchmarks provide comprehensive testbeds, reflecting real-world complexity and multilingual reach. Experimental and efficiency analyses reveal contrastive VLMs and two-stage retrieval pipelines as current state-of-the-art, with clear opportunities for progress in multilingual modeling, fine-grained semantics, and scalable architecture design [2512.20174, 2505.11651].

Source: https://www.emergentmind.com/topics/natural-language-based-document-image-retrieval-nl-dir