Self-Distilled RPN for Fine-Grained Visual Tasks

• The paper introduces a self-distillation approach that converts noisy middle-layer attention into precise, annotation-free RoI predictions.
• It employs a lightweight RPN architecture integrated on top of frozen pretrained layers to efficiently refine fine-grained features.
• Empirical results show significant accuracy improvements on OCR and document understanding tasks compared to traditional RoI methods.

Searching arXiv for the specified SD-RPN papers to ground the article in the latest literature. Self-Distilled Region Proposal Network (SD-RPN) is an annotation-free RoI localization module for multimodal LLMs (MLLMs) that distills the model’s own intermediate attention into a lightweight Region Proposal Network, enabling high-resolution fine-grained perception without requiring costly supervision or full model fine-tuning. It was introduced in the context of fine-grained MLLM perception in “Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception” and was subsequently incorporated as a core refinement branch in Q-Zoom, a query-aware adaptive perception framework for efficient MLLMs (Shi et al., 21 Sep 2025, Shi et al., 8 Apr 2026).

1. Problem formulation and design objective

SD-RPN addresses a central bottleneck in MLLM perception: fine-grained tasks such as reading small text, document understanding, and dense scene perception require high-resolution visual information, yet processing entire high-resolution images is computationally prohibitive. In the formulation used by LLaVA-like systems, the model processes low-resolution visual tokens and then decodes auto-regressively; a natural compromise is therefore to identify a small Region-of-Interest (RoI), crop and up-sample it, and feed that crop back into the model (Shi et al., 21 Sep 2025).

The immediate motivation for SD-RPN is the trade-off exhibited by prior RoI strategies. Training-based approaches depend on large-scale annotated datasets, while training-free methods that use internal attention are computationally inefficient and less accurate, requiring either multi-pass prefill stages or reliance on the slow auto-regressive decoding process. The SD-RPN formulation is explicitly designed to resolve this trade-off by transforming noisy middle-layer attention into pseudo-labels and then training a lightweight predictor that localizes RoIs in one forward pass. In Q-Zoom, the same idea is further embedded within a two-stage inference policy in which a lightweight Dynamic Gating Network first predicts whether high-resolution refinement is needed and activates SD-RPN only for queries that demand fine-grained perception (Shi et al., 8 Apr 2026).

This architecture suggests a broader conceptual shift from direct use of raw cross-attention to self-distilled localization. Rather than treating attention maps as final grounding outputs, SD-RPN treats them as imperfect teacher signals that can be denoised, sparsified, and distilled into a deployable module.

2. Self-distilled pseudo-label generation

The teacher signal in SD-RPN is derived from response-to-image cross-attention in the MLLM’s middle layers. In the original formulation, visual token features from layer $l$ are denoted $H_v\in\mathbb{R}^{(HW)\times d}$, and the cross-attention scores are $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$, softmaxed over visual tokens for each of the $N_t$ response tokens. Averaging over heads and response tokens yields a per-pixel RoI map $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$. In Q-Zoom, the same teacher-stage logic is expressed as a middle-layer grounding map $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$, obtained by averaging attention weights $A^l\in\mathbb R^{N_t\times HW}$ over text tokens (Shi et al., 21 Sep 2025, Shi et al., 8 Apr 2026).

A key problem is that raw attention is noisy. The papers identify attention “sinks” and incomplete activation as recurrent failure modes. SD-RPN therefore begins with sink-token removal. Visual token $j$ is suppressed when its feature norm exceeds a threshold $\tau_{\rm norm}$, giving

$$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$

After denoising, SD-RPN uses selective label assignment rather than dense supervision. Let $H_v\in\mathbb{R}^{(HW)\times d}$0 and $H_v\in\mathbb{R}^{(HW)\times d}$1. The foreground set is

$H_v\in\mathbb{R}^{(HW)\times d}$2

and the minimal enclosing bounding box around $H_v\in\mathbb{R}^{(HW)\times d}$3 is $H_v\in\mathbb{R}^{(HW)\times d}$4. The background set is

$H_v\in\mathbb{R}^{(HW)\times d}$5

The pseudo-label map is then tri-state: $H_v\in\mathbb{R}^{(HW)\times d}$6

The use of $H_v\in\mathbb{R}^{(HW)\times d}$7 to mark ignored tokens is central. It formalizes ambiguity resolution by excluding uncertain positions from supervision instead of forcing them into foreground or background. Q-Zoom reports that the best results are obtained at $H_v\in\mathbb{R}^{(HW)\times d}$8 and $H_v\in\mathbb{R}^{(HW)\times d}$9, and that tying them to the same value degrades performance by $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$0–$\mathbf A\in\mathbb{R}^{N_t\times(HW)}$1 points, indicating that asymmetric foreground/background criteria are materially important for the quality of the distilled labels (Shi et al., 8 Apr 2026).

3. Network architecture and training objective

SD-RPN is deliberately lightweight. In the original instantiation, the first $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$2 layers of the pretrained MLLM are frozen, and $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$3 trainable Transformer blocks are stacked on top, initialized from layers $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$4 to $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$5 of the MLLM. For LLaVA-1.5-7B, the reported example uses $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$6 frozen layers and $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$7 trainable RPN blocks; in Q-Zoom, all main experiments likewise use $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$8, and ablations on Qwen2.5-7B show that performance peaks at $\mathbf A\in\mathbb{R}^{N_t\times(HW)}$9 and dips at larger depths (Shi et al., 21 Sep 2025, Shi et al., 8 Apr 2026).

The predictor operates directly on intermediate hidden states. From the RPN’s last hidden layer, SD-RPN collects all visual tokens $N_t$0 and a set of RoI query vectors $N_t$1, where each query is the hidden state of the last token of a user-question turn. In Q-Zoom’s single-query setting, this is expressed as the hidden state of the final user query token, $N_t$2, together with the visual token states $N_t$3 (Shi et al., 21 Sep 2025, Shi et al., 8 Apr 2026).

Dense RoI prediction is produced by reusing Transformer attention projections. After normalization and linear projections,

$N_t$4

and the dense score map is

$N_t$5

Q-Zoom emphasizes that, instead of adding a new conv head or anchors, SD-RPN repurposes the $N_t$6-th block’s self-attention matrices $N_t$7; in practice, multi-head scores are computed per head and then averaged (Shi et al., 8 Apr 2026).

No explicit box-regression head is used. SD-RPN outputs a per-token RoI likelihood or dense spatial heatmap. In the original training setup, the student branch predicts $N_t$8 and is optimized against the pseudo-label map with a masked binary cross-entropy over non-ignored tokens: $N_t$9 Back-propagation is restricted to the $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$0 RPN layers, while the first $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$1 layers remain frozen. In Q-Zoom this same objective is written as $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$2 and described as selective binary cross-entropy, again restricted to tokens with $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$3 (Shi et al., 21 Sep 2025, Shi et al., 8 Apr 2026).

This design suggests that SD-RPN is less a standalone detector than a specialized distillation head for internal MLLM grounding. Its efficiency derives from weight reuse, frozen early computation, and sparse supervision rather than from external localization annotations.

4. Inference workflow and system-level integration

In the original pipeline, SD-RPN is inserted between coarse visual encoding and final answer generation. The operational sequence is: extract response-to-image cross-attention maps from the MLLM’s middle layers during a single auto-regressive pass; denoise and sparsify this raw attention to form a pseudo-label map of foreground, background, and ignored tokens; train a small RPN on top of frozen early layers of the MLLM to predict this pseudo-label map in one forward pass; and, at inference, run the RPN to get a dense RoI heatmap, threshold and extract a crop, up-sample that crop, re-insert its tokens, and finally decode the answer (Shi et al., 21 Sep 2025).

Q-Zoom makes the inference path more explicit as a conditional two-stage procedure. A lightweight Dynamic Gating Network first examines the coarse, low-resolution visual features plus the user query and predicts a binary “Need-Refine” flag. If the prediction is “No-Refine,” the model bypasses high-resolution processing entirely and generates an answer directly from the frozen backbone’s coarse features. If the prediction is “Need-Refine,” SD-RPN produces a dense spatial heatmap from the same intermediate hidden states; after sigmoid, Gaussian smoothing $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$4, and thresholding by $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$5, the minimal axis-aligned bbox is extracted, cropped from the original high-resolution image, re-encoded, and inserted between $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$6 and $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$7 at layer $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$8 (Shi et al., 8 Apr 2026).

Q-Zoom additionally introduces partial-prefill KV-cache reuse. Because coarse tokens $\mathbf M_{\mathrm{RoI}}\in\mathbb R^{H\times W}$9 up to layer $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$0 are identical between the first and second pass, their KV caches are stored. Only $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$1 are re-forwarded through layers $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$2, after which $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$3 are concatenated and processed through layers $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$4 (Shi et al., 8 Apr 2026).

A further systems issue is coordinate consistency after crop reinsertion. Q-Zoom identifies that local tokens lose their original global coordinate context and addresses this using continuous spatio-temporal alignment with Multimodal Rotary Positional Embeddings (MRoPE). RoI tokens are assigned a temporal offset $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$5 and interpolated spatial coordinates within the original box $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$6, so that the local crop lies on a distinct temporal plane while retaining the correct spatial coordinates. The same framework then applies targeted Post-Supervised Fine-Tuning on approximately $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$7K mined hard samples, freezing the vision encoder and projector and fine-tuning only the LLM with a standard next-token objective (Shi et al., 8 Apr 2026).

5. Data efficiency, training settings, and empirical behavior

The original paper emphasizes data efficiency. It reports that, despite being trained on only a few, for example $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$8K, question-answer pairs, SD-RPN demonstrates exceptional data efficiency and generalization, achieving over a $M_{\mathrm{RoI}}^l\in\mathbb R^{H\times W}$9 absolute accuracy improvement on unseen benchmarks including TextVQA, DocVQA, and V-Star. Training details for the LLaVA-1.5-7B example are AdamW, learning rate $A^l\in\mathbb R^{N_t\times HW}$0, batch size $A^l\in\mathbb R^{N_t\times HW}$1, $A^l\in\mathbb R^{N_t\times HW}$2 epoch, and a cosine schedule with $A^l\in\mathbb R^{N_t\times HW}$3 warm-up (Shi et al., 21 Sep 2025).

Q-Zoom provides a larger-scale account of pseudo-label generation and training. For the Qwen series, SD-RPN pseudo-labels use a total of approximately $A^l\in\mathbb R^{N_t\times HW}$4K samples: $A^l\in\mathbb R^{N_t\times HW}$5K GQA, $A^l\in\mathbb R^{N_t\times HW}$6K OCR-VQA, and $A^l\in\mathbb R^{N_t\times HW}$7K VCoT-DocVQA. LLaVA variants omit the $A^l\in\mathbb R^{N_t\times HW}$8K Document samples for $A^l\in\mathbb R^{N_t\times HW}$9K total. The optimizer is AdamW with weight decay $j$0, $j$1, $j$2, gradient clipping $j$3, cosine decay with linear $j$4 warm-up, peak learning rate $j$5 for SD-RPN and Gate, batch size $j$6, and training epochs $j$7 for each module (Shi et al., 8 Apr 2026).

A notable ablation concerns pseudo-label data size. On Qwen2.5-7B, even $j$8K distilled samples yield strong gains, with $j$9 average on OCR+V*, while the full $\tau_{\rm norm}$0K yields $\tau_{\rm norm}$1. A comparison “GT-box” baseline trained on $\tau_{\rm norm}$2K human boxes scores $\tau_{\rm norm}$3. This indicates that self-distilled pseudo-labels can be competitive with human box supervision at the reported scale (Shi et al., 8 Apr 2026).

Q-Zoom also reports that, when varying the backbone split $\tau_{\rm norm}$4 with $\tau_{\rm norm}$5, performance peaks at $\tau_{\rm norm}$6, described as the layer in which MLLM internal grounding is strongest. This finding reinforces the original SD-RPN premise that middle-layer attention provides the most useful supervisory signal, not merely a convenient one (Shi et al., 8 Apr 2026).

6. Performance profile, comparisons, and limitations

The original evaluation integrates SD-RPN into LLaVA-1.5 at both $\tau_{\rm norm}$7B and $\tau_{\rm norm}$8B scales and into DeepSeek-VL at $\tau_{\rm norm}$9B and $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$0B. Benchmarks include DocVQA, ChartQA, OCRBench, InfoVQA, TextVQA, V-Star Bench, POPE, and HR-Bench at $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$1K and $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$2K. Quantitatively, on Document/OCR, SD-RPN + LLaVA-1.5-7B raises average from $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$3 to $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$4 ($$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$5 points), and on V-Star Bench from $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$6 to $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$7 ($$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$8 points). The paper also states consistent $$(\mathbf M'_{\mathrm{RoI}})_j = \begin{cases} 0, & \Vert (H_v)_j\Vert_2 > \tau_{\rm norm},\ (\mathbf M_{\mathrm{RoI}})_j, & \text{otherwise.} \end{cases}$$9–$H_v\in\mathbb{R}^{(HW)\times d}$00 point absolute improvements across models and tasks, and Figure 1 shows cases where baseline LLaVA misses small text or objects while SD-RPN correctly crops and reads them (Shi et al., 21 Sep 2025).

Efficiency is a central part of the method’s profile. SD-RPN requires only a single forward pass through layers $H_v\in\mathbb{R}^{(HW)\times d}$01 of the MLLM to predict the RoI map, with no autoregressive decoding and no multi-pass prefills. In practice, RoI prediction is $H_v\in\mathbb{R}^{(HW)\times d}$02 faster than ViCrop’s cropping stage, and overall throughput relative to the full-image baseline is approximately $H_v\in\mathbb{R}^{(HW)\times d}$03 for LLaVA-1.5-7B, reported as significantly better than other two-stage methods (Shi et al., 21 Sep 2025).

Q-Zoom reframes this trade-off through conditional activation. On Qwen2.5-VL-7B, the baseline with $H_v\in\mathbb{R}^{(HW)\times d}$04-token max achieves $H_v\in\mathbb{R}^{(HW)\times d}$05 average on Document/OCR tasks at $H_v\in\mathbb{R}^{(HW)\times d}$06 throughput. Adding SD-RPN alone increases the average to $H_v\in\mathbb{R}^{(HW)\times d}$07 ($H_v\in\mathbb{R}^{(HW)\times d}$08 points) but drops throughput to $H_v\in\mathbb{R}^{(HW)\times d}$09; adding SD-RPN + Post-SFT gives $H_v\in\mathbb{R}^{(HW)\times d}$10 ($H_v\in\mathbb{R}^{(HW)\times d}$11) at $H_v\in\mathbb{R}^{(HW)\times d}$12; and the full Q-Zoom system, SD-RPN + SFT + Gate, gives $H_v\in\mathbb{R}^{(HW)\times d}$13 ($H_v\in\mathbb{R}^{(HW)\times d}$14) at $H_v\in\mathbb{R}^{(HW)\times d}$15. The paper attributes the throughput recovery to routing easy queries past the RoI branch (Shi et al., 8 Apr 2026).

Q-Zoom also reports a direct comparison with training-free RoI strategies on LLaVA-1.5-7B: raw cross-attention thresholding reaches $H_v\in\mathbb{R}^{(HW)\times d}$16 average at $H_v\in\mathbb{R}^{(HW)\times d}$17, GroundingDINO with one box reaches $H_v\in\mathbb{R}^{(HW)\times d}$18 at $H_v\in\mathbb{R}^{(HW)\times d}$19, GroundingDINO with two boxes reaches $H_v\in\mathbb{R}^{(HW)\times d}$20 at $H_v\in\mathbb{R}^{(HW)\times d}$21, and SD-RPN reaches $H_v\in\mathbb{R}^{(HW)\times d}$22 at $H_v\in\mathbb{R}^{(HW)\times d}$23, the best of these reported configurations (Shi et al., 8 Apr 2026).

The stated limitations are equally important. SD-RPN relies on the MLLM’s intrinsic attention; if the middle-layer attention is extremely poor for certain queries, pseudo-labels degrade. It has no explicit box-regression head, producing a heatmap rather than precise box coordinates. The two-stage crop-and-decode design still adds latency. The papers identify several possible extensions: a small regression head or IoU loss, adaptive token allocation such as early-exit on unambiguous regions, extensions to video or multi-frame inputs via self-distilling spatio-temporal masks, and joint learning of task and localization signals or the use of stronger teacher LLMs for pseudo-labels (Shi et al., 21 Sep 2025).

Taken together, these results position SD-RPN as a self-distilled RoI mechanism that converts noisy internal grounding into a fast, trainable predictor. A plausible implication is that its main significance lies not only in its reported gains on fine-grained VQA and document understanding benchmarks, but also in its demonstration that intermediate attention can serve as scalable supervision for localization modules in MLLMs when it is explicitly denoised and filtered.