---
title: 'UniDoc-RL: Hierarchical RL for Visual Document QA'
url: https://www.emergentmind.com/topics/unidoc-rl
type: topic
---

# UniDoc-RL: Hierarchical RL for Visual Document QA

Searching arXiv for UniDoc-RL and closely related RL/document-reasoning papers to ground the article with current citations.
UniDoc-RL is a unified reinforcement learning framework for coarse-to-fine visual retrieval-augmented generation in large vision-language models. It addresses document question answering and reasoning settings in which an agent must not only retrieve relevant document images from a large-scale visual corpus, but also refine those candidates, localize evidence inside them, and generate a final answer. The framework formulates visual information acquisition as a sequential decision-making problem with a hierarchical action space spanning retrieval, reranking, active visual perception, and reasoning, and it is trained with a dense multi-reward scheme under Group Relative Policy Optimization (GRPO) [2604.14967]. In this formulation, visual RAG is treated not as a fixed preprocessing pipeline but as an interactive Thought-Action-Observation process in which the model progressively suppresses irrelevant content and attends to information-dense regions [2604.14967].

## 1. Conceptual framing and problem setting

UniDoc-RL studies coarse-to-fine visual RAG for large vision-language models in a setting with a query \(Q\), a large-scale visual corpus \(\mathcal{C} = \{c_1, c_2, \dots, c_N\}\), and a target answer \(y\). The task is to retrieve relevant images, identify evidence inside them, and generate the final answer [2604.14967]. The motivating assumption is that visually rich documents such as slides, scanned pages, charts, tables, and reports require more than coarse retrieval. In this setting, successful reasoning depends on progressively refining evidence from document-level search to region-level inspection.

The framework is motivated by three stated difficulties in visual RAG. First, retrieval noise is more harmful because visual corpora contain dense content and much redundancy, so coarse similarity search often returns images that are only superficially related. Second, important information is local: a full page or slide may contain many irrelevant regions, so feeding the entire image to the model can waste context and obscure key details. Third, optimization suffers from credit assignment if training relies only on sparse final-answer rewards, because failures can originate in retrieval, selection, cropping, or reasoning [2604.14967].

This places UniDoc-RL in a different design space from text-only RAG or single-pass document QA. A plausible implication is that the framework treats evidence acquisition itself as the core learning problem rather than as a static upstream component. That emphasis aligns it with broader attempts to unify retrieval and reasoning through RL, although UniDoc-RL is specifically document-visual and action-hierarchical rather than general-purpose RAG-RL [2508.06165].

## 2. Hierarchical action space and Thought-Action-Observation dynamics

UniDoc-RL introduces a coarse-to-fine hierarchical action space with four explicit stages: Image Search, Precise Selection, Visual Perception, and Answer generation [2604.14967]. The interaction is modeled as a Thought-Action-Observation process in which, at each step \(t\), the policy \(\pi_\theta\) produces a thought \(T_t\) and action \(A_t\), receives observation \(O_t\), and repeats until it emits an answer.

In the Image Search stage, the model emits a search query using `<search> ... </search>`, and the environment performs retrieval:
\[
O_t = \text{Search}(q, \mathcal{C}).
\]
This returns a candidate pool of images or document pages [2604.14967].

In the Precise Selection stage, the model uses `<select> ... </select>` to choose relevant images from the retrieved candidates:
\[
O_{t+1} = \text{Select}(O_t, \mathcal{I}),
\]
where \(\mathcal{I}\) denotes selected indices. This stage is presented as LVLM-based reranking that narrows the semantic gap left by generic retrieval [2604.14967].

In the Visual Perception stage, the model performs active region-level perception with `<bbox> ... </bbox>`, and the environment crops or zooms the specified region:
\[
O_{t+2} = \text{VP}(O_{t+1}, R).
\]
This is the framework’s active perception mechanism: instead of passively consuming full images, the agent identifies an information-dense subregion and reasons over a higher-resolution crop [2604.14967].

When sufficient evidence has been collected, the model emits `<answer> ... </answer>` and terminates. The paper interprets this hierarchy as mirroring human information seeking: search broadens coverage, selection suppresses irrelevant hits, perception zooms into key regions, and reasoning extracts the answer [2604.14967].

The rollout procedure is iterative. The policy generates a response, the system parses the special action tags, executes one of search, select, crop, or answer, and injects the resulting observation back into the dialogue history as a new “User” message. This insertion strategy is explicitly justified as aligning the interaction format with the pretraining distribution of LVLMs, where visual inputs often appear in user prompts [2604.14967]. This suggests that UniDoc-RL is not only an RL architecture but also a prompt-format engineering choice designed to preserve compatibility with pretrained multimodal conversational behavior.

## 3. Reward design and GRPO optimization

A central contribution of UniDoc-RL is its dense multi-reward supervision. Rather than relying only on outcome reward from final answer correctness, the framework assigns stage-specific rewards for action syntax, retrieval quality, image selection, region localization, and answer quality. The total reward is
\[
r_{total} = \lambda_1 r_{pat} + \lambda_2 r_{ir} + \lambda_3 r_{sel} + \lambda_4 r_{crop} + \lambda_5 r_{ans},
\]
where \(r_{pat}\) is the pattern or format reward, \(r_{ir}\) is the image retrieval reward, \(r_{sel}\) is the selection reward, \(r_{crop}\) is the visual perception or cropping reward, and \(r_{ans}\) is the answer correctness reward [2604.14967].

The pattern reward checks whether the agent follows the required action syntax, including proper use of `<search>`, `<select>`, and `<bbox>` tags. The retrieval reward is defined with NDCG over the trajectory’s retrieved candidates:
\[
r_{ir} = NDCG(C_{trj}, C_{rel}),
\]
where \(C_{trj}\) is the merged ranked candidate list over retrieval steps and \(C_{rel}\) is the set of gold relevant images. The trajectory-level candidate list is defined by interleaving candidates from multiple retrieval rounds:
\[
C_{trj} = \{c_0^1, \ldots, c_0^m, c_1^1, \ldots, c_1^m, \ldots, c_k^1, \ldots, c_k^m\}.
\]
This reward is intended to encourage both retrieval of relevant evidence and favorable ranking of that evidence [2604.14967].

Selection reward is supervised by whether a chosen image belongs to the relevant set:
\[
r_{sel}^i =
\begin{cases}
1 & \text{if } c_{sel}^i \in C_{rel} \\
0 & \text{if } c_{sel}^i \notin C_{rel}
\end{cases}
\]
with
\[
r_{sel} = \frac{1}{m} \sum_{i=1}^{m} r_{sel}^i.
\]
The paper identifies a failure mode in which no gold image appears in a retrieved candidate set, making the reward uniformly zero. To address this, it uses a pseudo-supervision strategy that treats the top-ranked retrieval candidate \(c_0^i\) as a pseudo-positive target in such cases [2604.14967].

The crop reward is defined by intersection-over-union between predicted and annotated regions:
\[
r_{crop} = IoU(\mathcal{B}^*, \hat{\mathcal{B}}),
\]
where \(\mathcal{B}^*\) are ground-truth boxes and \(\hat{\mathcal{B}}\) are predicted boxes. The answer reward is scored by a model-based reward model,
\[
r_{ans} = \mathcal{RM}(A_{gen}, A_{ref}, Q),
\]
and the appendix specifies Qwen2.5-72B-Instruct as a binary judge producing 0/1 correctness [2604.14967].

Optimization uses GRPO. UniDoc-RL employs a two-stage pipeline consisting of full-parameter supervised finetuning as a warm start, followed by GRPO RL finetuning that optimizes the dense multi-reward objective [2604.14967]. The appendix reports a group size of 5, a KL loss coefficient of 0.01, actor learning rate \(1\times 10^{-6}\), one RL epoch, max prompt length 40000, and max response length 1024 [2604.14967]. The paper emphasizes GRPO’s appeal in avoiding a separate value network.

This reward design is notable against a broader methodological backdrop. A critique of many GRPO-style LLM RL systems is that common LLM-as-MDP assumptions can collapse RL into outcome-driven supervised learning when states are token prefixes and terminal reward is uniformly distributed across tokens [2505.13697]. UniDoc-RL’s action space and reward structure differ materially from that degenerate setup: it supervises retrieval, selection, and cropping as distinct subtasks with dense rewards tied to external tool interaction and annotated intermediate actions [2604.14967]. This suggests that UniDoc-RL is closer to a genuine sequential control problem than outcome-only RL post-training.

## 4. Data curation and annotated reasoning trajectories

UniDoc-RL relies on a curated dataset of reasoning trajectories with fine-grained action annotations. The source datasets are SlideVQA, DoubleBench, VisR-Bench, DocBench, and DUDE, covering slides, PDFs, scanned documents, multilingual settings, tables, charts, figures, and long-context document understanding [2604.14967].

Trajectory synthesis is performed by a strong teacher model, Qwen3-VL-235B, sampling
\[
(T_t, A_t) \sim \pi_{teacher}(\cdot \mid H_{t-1}).
\]
The teacher generates search, selection, and crop decisions. For visual perception, the paper describes a more involved pipeline: layout parsing with Mineru generates candidate bounding boxes, the teacher chooses the most useful region among those candidates, and that region is stored as the ground-truth crop action [2604.14967]. This procedure is intended to produce crop annotations more accurate than naive box generation.

The filtering pipeline contains three components. Quality filtering discards trajectories whose final answer is wrong. Difficulty-aware filtering retains examples that are not trivial for an intermediate SFT model. RL data curation keeps challenging but solvable samples where retrieval works but reasoning or perception can still fail [2604.14967]. The paper states that this matters because RL works best on data where improvement is possible but not too easy.

The final curated dataset contains **12,621 samples for SFT** and **5,537 samples for RL** [2604.14967]. The SFT breakdown is SlideVQA 8271, DoubleBench 1274, VisR-Bench 1518, DocBench 657, and DUDE 901. The RL breakdown is SlideVQA 2613, DoubleBench 993, VisR-Bench 1630, DocBench 301, and DUDE not used for RL in the table [2604.14967].

This dataset design is important for interpreting the method. UniDoc-RL is not trained from scratch on undifferentiated reward signals; it is initialized with structured trajectories that expose the full action hierarchy. A plausible implication is that the framework depends heavily on annotation quality and curriculum design, which the paper later acknowledges indirectly through its practical caveats [2604.14967].

## 5. Experimental setup, baselines, and quantitative results

The framework is evaluated on three benchmarks: SlideVQA, ViDoSeek, and MMLongBench. SlideVQA includes single-hop and multi-hop slide reasoning; ViDoSeek emphasizes extraction and logic on visually rich documents; MMLongBench focuses on long-context document understanding with strong reliance on visual content [2604.14967]. Experiments use two LVLM backbones: Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct [2604.14967].

The baseline set includes Vanilla RAG, ReAct, Search-R1(-VL), and VRAG-RL. The paper distinguishes OCR-based RAG methods from purely visual RAG methods and uses an external evaluation model returning binary correctness, with accuracy as the main metric [2604.14967].

On Qwen2.5-VL-3B-Instruct, the paper reports overall accuracy of **53.5** for VRAG-RL and **71.0** for UniDoc-RL [2604.14967]. On Qwen2.5-VL-7B-Instruct, it reports **57.1** for VRAG-RL and **74.8** for UniDoc-RL [2604.14967]. The paper highlights gains of up to **17.5%** on the 3B model and **17.7%** on the 7B model over the strongest prior RL baseline [2604.14967]. It states that UniDoc-RL consistently surpasses state-of-the-art baselines across all benchmarks and model sizes.

The interpretation offered in the paper is threefold. OCR-only pipelines are weaker because they lose layout and spatial cues. Purely visual retrieval is better than OCR-only retrieval. RL with hierarchical actions performs best because it jointly improves retrieval quality, candidate filtering, local evidence extraction, and final reasoning [2604.14967].

In relation to neighboring work, UniDoc-RL differs from CogDoc, which also studies unified document reasoning but organizes it as “Fast Reading” and “Focused Thinking” and reports that direct RL can outperform SFT+RL because SFT induces “policy conflict” between global localization and local grounding [2512.12658]. UniDoc-RL instead uses SFT warm start plus GRPO finetuning and a more explicit action hierarchy of search, select, crop, and answer [2604.14967]. The difference suggests that “unified document RL” is not a single design pattern: some systems unify coarse and fine reasoning through mode switching, whereas UniDoc-RL unifies them through executable hierarchical actions.

## 6. Ablations, behavioral analysis, and limitations

The action-space ablation studies three variants: no selection and no perception, selection only, and selection plus perception. On Qwen2.5-VL-3B-Instruct, the reported scores are **66.6**, **70.0**, and **71.0**, respectively [2604.14967]. The stated conclusion is that both actions help, with selection giving the larger boost on reasoning-heavy tasks and perception helping most on detail-heavy tasks.

The reward ablation incrementally adds retrieval reward, selection reward, and visual perception reward on top of a vanilla sparse-reward setup. The best results come from the full dense setup, and the paper interprets each reward as aligned with its corresponding subtask: retrieval reward improves candidate quality, selection reward improves semantic filtering, and perception reward improves fine-grained localization [2604.14967].

The paper also reports behavioral analyses beyond end-task accuracy. Adding the selection step increases retrieval hit rate of ground-truth images from **79.7% to 85.0%** on SlideVQA, from **74.8% to 85.7%** on ViDoSeek, and from **48.9% to 52.7%** on MMLongBench [2604.14967]. It further states that SFT models are conservative, using crop actions too rarely and often defaulting to passive full-image reading, whereas after RL training crop frequency rises and becomes closer to the teacher’s behavior [2604.14967]. Qualitative examples reportedly show that SFT often produces trivial crops close to the full image, while UniDoc-RL learns more precise crops around the true region of interest [2604.14967]. Case studies indicate adaptive strategy: crop when evidence is small and hidden, skip crop when the selected image is already readable [2604.14967].

These observations are significant because they suggest that the method changes action behavior rather than merely answer distribution. In that respect, UniDoc-RL resembles broader RL systems that seek to learn when to invoke tools, not only how to produce final responses. UR\(^2\), for example, frames retrieval as a dynamic policy choice and argues that the model should learn to reason directly when it can and search when it must [2508.06165]. UniDoc-RL instantiates a comparable principle in the visual-document domain, extending the tool-use policy down to image reranking and crop selection [2604.14967].

The paper does not foreground limitations, but several are stated or implied. The method depends on high-quality action annotations and curated trajectories. It uses external tools such as search and layout parsing, so performance is partly tool-dependent. The action space and reward design are task-specific and may require re-annotation or reward redesign for new domains. The system relies on a strong reward model for answer correctness, which can introduce evaluation bias. The crop reward assumes access to ground-truth regions during training, which is not always available in real applications [2604.14967].

Taken together, UniDoc-RL presents visual RAG as a hierarchical RL control problem in which retrieval, semantic filtering, active perception, and reasoning are jointly optimized. Its main empirical claim is that this integration improves retrieval recall, crop precision, tool-use behavior, and final accuracy relative to prior visual RAG and RL baselines [2604.14967]. A plausible broader implication is that document reasoning systems benefit when intermediate evidence-acquisition steps are made explicit, executable, and directly rewarded, rather than treated as latent or purely prompt-induced behavior.

Source: https://www.emergentmind.com/topics/unidoc-rl