Where Grounding Accuracy Lives on the IoU Curve: Label-Free Inference-Time Boundary Refinement

Abstract: Vision--LLMs can identify the correct referent while returning an imprecise bounding box. We study whether a frozen direct-answer model can use its own prediction to allocate one additional localized observation without accessing target annotations at inference. Label-free precision refinement (LFPR) routes predicted-small regions to a higher-resolution pass, re-grounds the expression inside a context crop, admits a candidate only under fixed geometric guards, and returns a fixed coordinate-wise midpoint. We report results across three evidence tiers. On 31,921 retrospective Ref-L4 expressions, LFPR raises mAcc$_{0.5:0.95}$ from 72.947\% to 76.013\% ([email protected] 88.531\%$\to$89.725\%, [email protected] 55.788\%$\to$61.142\%). A frozen transfer to 30,969 RefCOCO/RefCOCO+/RefCOCOg expressions improves every dataset at [email protected], mAcc, and mean IoU (pooled mAcc $+0.645$, [email protected] $+0.817$), while [email protected] is unchanged overall: routing alone gains $+1.162$ points there, but crop, guards, and fusion give back $-1.192$, offsetting rather than showing no strict-IoU effect. A prospective, image-disjoint Flickr30K Entities evaluation improves every endpoint (mAcc $+0.973$, [email protected] $+1.022$), more strongly under a single-box variant (mAcc $+2.575$, [email protected] $+3.689$). The same operator applied to two released grounding specialists improves every endpoint ([email protected] $+1.569$/$+6.716$ for EGM-4B/8B) at roughly twice the latency, composing with specialist training rather than replacing it. A genuine unguarded control (guard removed from the same candidates) underperforms the incumbent on every metric, showing the guard is load-bearing. Together, these results show that referent selection and boundary precision are partially separable, with different components moving opposing regions of the IoU curve -- behavior a single threshold cannot reveal.

• {
• The paper introduces a label-free, inference-time method that improves boundary precision in frozen vision-language models, increasing accuracy in referring expression grounding tasks.
• The LFPR approach uses a label-free resolution router and context crop to re-ground expressions, which improves baseline accuracy particularly at strict IoU thresholds (mAcc) without querying ground truth annotations; proposing a three-stage pipeline.
• Analysis shows that this method provides proposing method combination makes specific strides in intermediate IoU thresholds, validating the refutation of the usual metric, mean-accuracy

Overview

This paper studies whether a frozen vision–LLM (VLM) can improve the boundary precision of its own referring-expression grounding predictions at inference time, without any weight update and without access to ground-truth annotations (2608.19553). The proposed operator, label-free precision refinement (LFPR), is deliberately narrow: it uses the model's first-pass box to decide when to spend additional computation, re-grounds the expression inside a context crop, admits the candidate only under fixed geometric guards, and fuses via a coordinate-wise midpoint. The central empirical claim is that referent selection and boundary placement are partially separable capabilities whose effects move opposing regions of the IoU threshold curve — a structure that single-threshold reporting such as [email protected] cannot reveal.

Method

LFPR operates on a frozen direct-answer incumbent $b_0 = f_\theta(I,q)$ in three stages. First, a label-free resolution router buckets the predicted box by pixel area ($s < 128$ px = small) and routes only small-bucket rows to a second full-image inference with a raised minimum-pixel budget of 4,194,304. The bucket is computed from $b_0$, not the annotation; it agrees with the annotation-derived bucket on ~89% of rows. Second, for every valid incumbent, a context crop of twice the incumbent's width and height ($\gamma=2.0$) is re-grounded by the same frozen checkpoint. Third, the crop candidate $b_c$ is admitted only if three geometric guards pass: zoom-consistency distance $\leq 0.35$, IoU$(b_c,b_0') \geq 0.25$, and area ratio in $[0.5, 2.0]$. Admitted candidates are fused as the midpoint $\widehat{b} = \tfrac{1}{2}b_0' + \tfrac{1}{2}b_c$; rejected candidates retain the pre-crop box. All constants were fixed before the final runs, and no step consults the ground-truth box.

The cost profile is modest: two forward passes for roughly four of five rows and three for routed rows, roughly doubling latency relative to the incumbent.

Evidence design

The paper stratifies evidence into three tiers with explicit epistemic status: a retrospective Ref-L4 analysis (31,921 expressions) whose published test distribution informed method development; a frozen cross-dataset transfer to the RefCOCO family (30,969 expressions) where nothing was retuned; and a prospective, image-disjoint Flickr30K Entities confirmation (14,481 rows under the standard merged-box protocol). All comparisons are paired on identical rows with image-cluster bootstrap intervals (10,000 resamples), Holm correction within declared families on the transfer analyses, and invalid outputs retained in every denominator. The authors candidly withdraw an earlier "test firewall" claim: pilot rows were reused and test-distribution diagnostics shaped development, so all Ref-L4 intervals describe sampling variability under a frozen comparison rather than confirmatory error rates.

Headline results

On Ref-L4, LFPR raises mAcc$_{0.5:0.95}$ from 72.947% to 76.013%, [email protected] from 88.531% to 89.725%, and [email protected] from 55.788% to 61.142%. On the frozen RefCOCO-family transfer, every dataset improves at [email protected], mAcc, and mean IoU (pooled +0.817 and +0.645 points respectively), but [email protected] is unchanged overall ($s < 128$0, adjusted $s < 128$1). The prospective Flickr30K merged-box evaluation improves every endpoint (mAcc +0.973, [email protected] +1.022), more strongly under the historical single-box protocol (+2.575 mAcc, +3.689 [email protected]).

Component decomposition: opposing stages

The most informative contribution is the same-row decomposition enabled by storing candidates before gating. On the transfer split:

• Resolution routing alone gains $s < 128$2 points at [email protected], concentrated exactly on the small bucket ($s < 128$3 points there versus negligible movement on large targets), confirming the mechanism the router assumes.
• Guarded crop re-grounding alone is also boundary-precision-positive ($s < 128$4 at [email protected]) but nearly flat at [email protected].
• Guards plus midpoint fusion give back strict-IoU accuracy ($s < 128$5 relative to the routed arm) in exchange for the largest [email protected] gain of any arm.
• A genuine unguarded ablation — recomputed from CRG's own stored candidates with the guard gate removed — underperforms the incumbent on every metric ($s < 128$6 at [email protected], $s < 128$7 at [email protected]), establishing that the guard is load-bearing rather than a discarded trade-off. A confidence-gated control on the same admission point fails to recover even the guard's modest [email protected] gain, supporting the specifically geometric criterion.

A guard×fusion factorial further shows that guarded replacement significantly beats the shipped midpoint policy at [email protected] and at the paper's own primary endpoint (mAcc), while losing at [email protected]/0.75; the authors retain midpoint fusion because it was fixed before the factorial was computed, and report the direct contrast transparently. Notably, this trade is dataset-dependent: on Flickr30K the guarded-midpoint cell dominates guarded replacement on every metric, so the Ref-L4 pattern is not universal.

Specialists and the redistribution thesis

Against two released grounding specialists (EGM-4B/8B) evaluated zero-shot on identical rows, the headline reading is unfavorable: a released 4B model doing one forward pass exceeds the refined 8B pipeline by 0.694 points at [email protected]. But the ordering inverts at strict thresholds — the guarded system beats the 8B specialist by $s < 128$8 points at [email protected] while being indistinguishable from the 4B specialist there. Scaling the specialist from 4B to 8B raises [email protected] by $s < 128$9 yet lowers [email protected] by $b_0$0: among base systems, the [email protected] model has the worst localization quality. This supports the paper's methodological recommendation that referring-expression results be reported as full threshold profiles or with mean accuracy alongside [email protected].

Crucially, applying LFPR unchanged to each specialist's own predictions improves every endpoint for both specialists ([email protected] $b_0$1/$b_0$2 for EGM-4B/8B) at roughly $b_0$3 latency, showing the operator composes with specialist training rather than competing with it. An image-level firewall audit finds zero exact-image overlap between specialist training data and evaluation partitions, though the authors correctly note this does not rule out distributional similarity effects.

Negative controls

The claim is bounded by an unusually thorough set of closed alternatives: supervised coarse-to-fine adaptation, full-parameter fine-tuning, iterative unguarded zooming, random/center-crop placement controls, matched self-consistency sampling, backbone scaling to 32B, and cross-family checkpoints all fail or regress. Placement controls collapse to ~42–44% [email protected], confirming the crop's gain requires an incumbent-informed window, not merely a second look. Two errata are documented openly: a silently discarded pixel-budget override that initially made routing a byte-for-byte no-op, and a Mean-IoU column transcribed against the wrong arm — both corrected with hash-verified artifacts.

Limitations

Five boundaries are stated plainly. The Ref-L4 estimate is retrospective, with program-level adaptation to the published test distribution and no external preregistration. The specialist comparison is only partly compute-normalized and is supervised-versus-zero-shot, hence not like-for-like despite the firewall audit. Operator-effect evidence covers one architecture family; cross-family checkpoints were screened only at the direct-answer stage after two candidates failed screening. Finally, the size threshold, guards, and fusion weight are fixed constants — a grid sweep found no Pareto-dominating configuration, but learning them would change the contribution's character. On Flickr30K, routing alone is actually the accuracy-maximizing point on the cost frontier, so the marginal value of crop/guard/fusion beyond routing is dataset-dependent.

Conclusion

The paper establishes that a frozen VLM can convert its own first prediction into a targeted second observation that improves grounding precision without labels, and that the resulting gains are unevenly distributed across the IoU curve: selective resolution routing robustly buys strict-IoU accuracy, while guarded crop fusion primarily protects permissive-threshold accuracy against an unfiltered second opinion. The honest accounting of retrospective selection, negative controls, and internal errata makes the positive claims narrower but considerably more credible than typical inference-time-refinement reports. Open questions include whether the router should extend beyond the small bucket (the medium bucket showed large strict-IoU gains once the floor took effect), whether learned guard thresholds would change the trade-off structure, and how the operator behaves across architecture families beyond Qwen3-VL.