- The paper introduces the GLaQ framework, which significantly improves multimodal reasoning through the effective use of visual evidence, as shown by performance improvements of 5.99 to 9.66 points across five benchmarks (V*, HRBench-4K/8K, MME-RealWorld-Lite, MMVP).
- GLaQ replaces autoregressive latent rollout with a fixed block of $K = 16$ learnable query slots, contextualizing queries directly in source visual tokens through the Latent Query Former, then reinjecting grounded visual represented slots as soft prompts.
Motivation and problem statement
Chain-of-thought reasoning in multimodal LLMs (MLLMs) verbalizes intermediate decisions as text, which compresses fine-grained visual details before they can be reused. Thinking-with-images methods address this by cropping, zooming, or annotating the image at inference time, but require predefined tools and additional external processing. Continuous visual latent reasoning offers an internal alternative by carrying intermediate computation in hidden states, yet the prevailing construction is autoregressive: each latent state conditions on its predecessors. The authors present similarity analyses showing that autoregressive latent tokens become increasingly alike over the sequence, forming a high-similarity block late in the trajectory — evidence that later states may repeat shared content rather than contribute complementary visual information. This observation motivates the central design question: can latent reasoning retain its compactness while grounding each state directly in source visual tokens and coordinating states without a sequential dependency chain?
The GLaQ framework
GLaQ replaces autoregressive latent rollout with a fixed block of K learnable query slots processed through a contextualize–ground–reinject pathway. When the MLLM emits a latent_start token, all K slots are inserted jointly and contextualized in one forward pass under the current image–text prefix. A lightweight Latent Query Former (LQFormer) — a two-layer Q-Former-style adapter with cross-attention to visual tokens, self-attention across slots, and FFN refinement — grounds these contextualized states against the original visual tokens V. The grounded block is then reinjected as soft visual prompts for a second decoding pass that resumes standard autoregressive text generation. Only the query slots interact with the N visual tokens, so the visual encoder and MLLM decoder remain unmodified.
The pairwise cosine-similarity structure of GLaQ's grounded queries qualitatively resembles that of auxiliary image tokens rather than late autoregressive latent states, supporting the claim that coordinated query slots retain differentiated information without a sequential chain.
Training strategy
Supervised training combines two objectives with weight λlatent​=0.3: a latent-alignment loss and next-token prediction on surrounding text (excluding internal continuous slots). Latent query self-distillation follows an asymmetric online–target design inspired by BYOL: an EMA teacher LQFormer grounds the same contextualized queries in a paired region-of-interest (ROI) view, while the student grounds them in the full image; slot-wise squared-error alignment transfers localized evidence to the inference-time full-image pathway without reconstructing dense visual tokens. Because both paths share identical ordered query blocks, slots correspond directly. SFT proceeds in two stages: 80K Visual-CoT samples (filtered to cases where the base model fails on the original image but succeeds with an auxiliary view), followed by 30K Zebra-CoT interleaved trajectories that teach when to interleave grounded-query blocks with text.
Reinforcement learning adopts Decoupled Policy Optimization (DePO) from HyLaR, treating post-LQFormer queries as continuous actions alongside categorical text tokens. DePO uses von Mises–Fisher density ratios, scaled-cosine KL terms, separate clipping ranges, and GRPO-style group-normalized advantages under a simple accuracy-plus-format reward (λfmt​=0.1); notably, no dedicated latent-reasoning reward is added.
Main results
Across five benchmarks (V∗, HRBench-4K/8K, MME-RealWorld-Lite, MMVP), GLaQ-7B improves its Qwen2.5-VL-7B backbone by 5.99–9.66 points and ranks first among all compared visual latent methods (LVR, SkiLa, Monet, HyLaR) on every benchmark. Representative gains over the backbone include +11.84 on V∗ spatial questions, +10.00 on HRBench-8K single-instance questions, +10.53 on MME-RealWorld-Lite reasoning, and +9.66 on MMVP. GLaQ also remains competitive with thinking-with-images systems (ZoomEye, Thyme, DeepEyes) while using only original images.
A resolution-dependent contrast emerges on HRBench: from 4K to 8K, gains on single-instance questions grow from 6.75 to 10.00 points, while cross-instance gains shrink from 5.25 to 3.25. This suggests grounded queries are most effective at recovering localized evidence degraded in high-resolution representations, whereas integrating evidence across multiple instances remains harder — a limitation the authors acknowledge directly.
Ablations and sensitivity
Component ablations isolate where the gains originate. GLaQ-SFT consistently beats a text-only SFT baseline trained on the same data, but removing the EMA teacher eliminates most of this advantage, and — more strikingly — bypassing LQFormer drops performance below text-only SFT on all five benchmarks even though first-pass query states are still reused as soft prompts. This controlled contrast indicates continuous query states alone are insufficient; the benefit depends on renewed, context-conditioned access to source visual tokens. On the RL side, text-only GRPO already yields substantial gains, but DePO adds a further 0.52–1.37 points on every benchmark under matched data and rewards, showing grounded queries function as optimizable latent actions rather than passive representations.
Hyperparameter studies reveal non-monotonic behavior on both fronts. λlatent​=0.3 ranks first on four of five benchmarks, while λlatent​=1.0 underperforms throughout, indicating self-distillation works best as an auxiliary regularizer that can otherwise overconstrain the student toward ROI-conditioned teacher representations. Query capacity shows a similar trade-off: K0 wins on all five benchmarks, but K1 falls below K2 on both HRBench settings and MMVP, plausibly because unconstrained slots attending to the same token pool introduce redundancy that dilutes the grounded signal. The authors caution this trend should not be read as test-time scaling, since larger K3 widens a single lookup rather than adding observations or reasoning steps.
Efficiency
On VK4 throughput measurements (191 examples, wall-clock including all visual operations), GLaQ advances the accuracy–efficiency frontier relative to three model families: versus its own backbone, DeepEyes, and HyLaR it improves accuracy by 8.90, 2.09, and 4.19 points while increasing throughput by 7.62%, 20.11%, and 5.70%, respectively. Because each query block has fixed capacity, grounding overhead stays bounded, unlike variable-length latent rollouts whose cost grows with trajectory length.
Limitations and open questions
The paper concedes several constraints explicitly. Evaluation covers only a single 7B backbone (Qwen2.5-VL-7B-Instruct), so scaling behavior of grounded latent queries is untested. Query capacity is fixed and non-adaptive; the non-monotonic K5 results suggest adaptive query allocation could help but is not explored. Supervision depends on auxiliary localized views for self-distillation, raising dependence on the availability and quality of ROI pairs, and the authors propose reducing this reliance as future work. The declining cross-instance gains at higher resolution leave open how grounded lookup can support multi-instance evidence integration. Finally, the qualitative similarity analysis motivating the method is not backed by a quantitative redundancy metric comparing grounded versus autoregressive latents.
Conclusion
GLaQ reformulates continuous visual latent reasoning as context-conditioned visual lookup: fixed query blocks are contextualized, grounded directly in source visual tokens via a lightweight adapter, and reinjected as soft prompts. Combining ROI-guided self-distillation with DePO yields consistent gains of 5.99–9.66 points over its backbone and first place among compared visual latent methods across five benchmarks, with simultaneous accuracy and throughput improvements. The ablation evidence supports the paper's central claim that explicit source-image grounding, rather than insertion of continuous states per se, drives the benefit — while leaving backbone scaling, adaptive capacity, and multi-instance integration as open problems.