---
title: Adaptive Visual In-document Retrieval (AVIR)
url: https://www.emergentmind.com/topics/adaptive-visual-in-document-retrieval-avir
type: topic
---

# Adaptive Visual In-document Retrieval (AVIR)

Adaptive Visual In-document Retrieval (AVIR) is a retrieval-augmented pipeline for efficient and accurate multi-page document question answering. It addresses the computational and attention challenges inherent in long-form visual question answering (VQA) by scoring and adaptively selecting document pages most relevant to a given question, subsequently leveraging a frozen large vision-language model (LVLM) for answer generation. AVIR reduces the average number of processed pages by approximately 70% compared to standard end-to-end approaches, while maintaining or surpassing state-of-the-art accuracy across multiple VQA benchmarks [2601.11976].

## 1. Framework Overview

AVIR operates through a multi-stage retrieval and selection design. The system begins with a lightweight page retrieval model that assigns relevance probabilities to each page with respect to the question. Pages are then selected using an adaptive clustering and thresholding routine: for longer documents, k-means clustering with $k=2$ partitions the pages by score; for shorter documents, a simple thresholding strategy is used. The final subset of relevant pages is fed into a frozen LVLM, which generates answers without additional fine-tuning.

A plausible implication is that AVIR’s modularity allows easy integration with new LVLM architectures, where only the retrieval head requires adaptation per dataset or domain.

## 2. Lightweight Page Retrieval and Scoring

### Architecture

- **Encoder**: The visual backbone uses a frozen Pix2Struct image-and-text encoder ($\approx$1.5B parameters), processing each document page $p_i$ rendered at 512×512 pixels and concatenated with the question text $Q$.
- **Hidden State Extraction**: Output is a sequence of hidden states $H_i \in \mathbb{R}^{T \times d}$, including a special [CLS] token embedding $h_i \in \mathbb{R}^{d}$ per page.
- **Scoring Head**: On top of $h_i$, two self-attention layers (dimension $d$), followed by a linear layer and sigmoid activation, yield relevance probability $r_i \in (0,1)$:
$$
a_i = \mathrm{LayerNorm}(h_i) \\
z_i = \mathrm{SelfAttn}_2(\mathrm{SelfAttn}_1(a_i)) \\
r_i = \sigma(w^\top z_i + b)
$$
where $\sigma(\cdot)$ denotes the sigmoid function. The scoring head adds approximately 100M trainable parameters (∼4% of the LVLM).

### Training

Binary labels $y_i \in \{0,1\}$ indicate whether a page contains the answer span. The loss function sums binary cross-entropy over pages:
$$
L = -\sum_{i=1}^n \left[ y_i \log r_i + (1-y_i) \log (1-r_i) \right]
$$

## 3. Adaptive Page Selection

AVIR’s selection algorithm adapts to document length and score distribution.

### Long Documents ($n \geq 4$)

1. **Clustering**: Apply k-means ($k=2$) on scores $\{r_i\}$, yielding centroids $\mu_0 < \mu_1$.
2. **Relevant Cluster Formation**: Pages assigned to the cluster with centroid $\mu_1$ form the candidate set $P_{rel} = \{ p_i \mid c_i = 1 \}$.
3. **Top-$K_{max}$ Filtering**: If $|P_{rel}| > K_{max}$ (with $K_{max}=8$), select the top $K_{max}$ pages by descending $r_i$.

### Short Documents ($n < 4$)

A fixed relevance-probability threshold $T = 0.6$ selects pages:
- If $\max_i r_i \geq T$, choose $P_{selected} = \{p_i \mid r_i \geq T\}$.
- Otherwise, select all pages.

#### Pseudocode

```python
def AdaptivePageSelector(P, R, T=0.6, K_max=8):
    n = len(P)
    if n < 4:
        if max(R) >= T:
            return [p_i for p_i, r_i in zip(P, R) if r_i >= T]
        else:
            return P
    else:
        # k-means on R with k=2
        c_i, mu_0, mu_1 = kmeans(R, k=2)
        R_rel = [p_i for p_i, c in zip(P, c_i) if c == (0 if mu_1 < mu_0 else 1)]
        if len(R_rel) > K_max:
            R_rel = sorted(R_rel, key=lambda p: R[P.index(p)], reverse=True)[:K_max]
        return R_rel
```

## 4. LVLM Integration for Answer Generation

Selected pages (typically 2–4 per query) are rendered and packed into the input stream to a frozen Qwen2.5-VL-3B model, quantized with AWQ. The final input consists of the text question prefixed (“Answer the question based on the following pages. Question: …”) and pixel grids of each selected page, maintaining original reading order. The LVLM receives no parameter updates; all adaptation occurs in the retrieval component. Cross-modal attention in the LVLM enables contextual reasoning over the compressed document subset and answer generation proceeds via token-wise autoregressive decoding.

This suggests that the separation between retrieval and answer models can decouple computational costs for context expansion from model parameter adaptation.

## 5. Empirical Results and Benchmark Performance

AVIR’s effectiveness has been evaluated on MP-DocVQA, SlideVQA, and DUDE datasets, with notable reductions in processed pages and improvements in prediction accuracy metrics.

- **MP-DocVQA**
    - Average processed pages per query: 2.5 (vs. 8.3 for end-to-end).
    - Top-1 Page Prediction Accuracy: 81.55%.
    - Average Normalized Levenshtein Similarity (ANLS): 0.8458.
- **SlideVQA**
    - Average processed pages per query: 2.9 (vs. 20.0 for end-to-end).
    - Exact Match (EM): 60.3%.
    - F1: 68.9%.
- **DUDE**
    - Overall ANLS: 0.4905 (Extractive: 0.6754, Abstractive: 0.6404).

### Comparative Tables

#### MP-DocVQA Results

| Method              | Params | Page-Pred (%) | ANLS   |
|---------------------|--------|---------------|--------|
| SelfAttnScoring     | 273 M  | 81.55         | 0.6199 |
| M3DOCRAG (8 B)      | —      | 81.05         | 0.8444 |
| Qwen2.5-3B-AWQ      | —      | —             | 0.8405 |
| AVIR (ours)         | 3 B    | 81.55         | 0.8458 |

#### SlideVQA Results

| Model           | Params | EM   | F1   |
|-----------------|--------|------|------|
| Qwen2.5-3B-AWQ  | 3 B    | 56.6 | 65.8 |
| AVIR (ours)     | 3 B    | 60.3 | 68.9 |

#### Top-K Ablation (SlideVQA)

| Strategy             | Avg Pages | EM   | F1   |
|----------------------|-----------|------|------|
| No Retrieval (20 pg) | 20.0      | 56.6 | 65.8 |
| Top-K=1              | 1.0       | 52.7 | 60.2 |
| Top-K=2              | 2.0       | 56.5 | 64.6 |
| Top-K=4              | 4.0       | 58.3 | 66.8 |
| Top-K=8              | 8.0       | 57.2 | 65.7 |
| APS (ours)           | 2.9       | 60.3 | 68.9 |

A plausible implication is that adaptive selection strategies allow for more efficient context compression than naive top-$K$ methods while retaining or improving predictive accuracy.

## 6. Significance and Practical Considerations

AVIR’s design mitigates computational bottlenecks and challenges posed by long documents in visual question answering, notably reducing the cost of LVLM inference without model fine-tuning. The retrieval module’s light footprint (∼100M parameters over a frozen LVLM) means adding AVIR does not require retraining large-scale multimodal models. Empirical performance demonstrates that adaptively selected context windows outperform fixed-length alternatives, delivering robust gains on multi-page scientific and presentation-style document benchmarks.

The modularity and dataset-specific retrieval adaptation suggests AVIR is applicable to other visual document analysis tasks requiring selective context curation for downstream language reasoning.

Source: https://www.emergentmind.com/topics/adaptive-visual-in-document-retrieval-avir