Test-Time Computing for Referring Multimodal Large Language Models

Abstract: We propose ControlMLLM++, a novel test-time adaptation framework that injects learnable visual prompts into frozen multimodal LLMs (MLLMs) to enable fine-grained region-based visual reasoning without any model retraining or fine-tuning. Leveraging the insight that cross-modal attention maps intrinsically encode semantic correspondences between textual tokens and visual regions, ControlMLLM++ optimizes a latent visual token modifier during inference via a task-specific energy function to steer model attention towards user-specified areas. To enhance optimization stability and mitigate language prompt biases, ControlMLLM++ incorporates an improved optimization strategy (Optim++) and a prompt debiasing mechanism (PromptDebias). Supporting diverse visual prompt types including bounding boxes, masks, scribbles, and points, our method demonstrates strong out-of-domain generalization and interpretability. The code is available at https://github.com/mrwu-mac/ControlMLLM.

• The paper introduces ControlMLLM++, a test-time adaptation framework that injects learnable visual prompts to achieve fine-grained region-level grounding in pre-trained MLLMs.
• It employs optimization of latent visual variables and targeted manipulation of cross-attention maps to steer model focus toward user-specified regions.
• Empirical results demonstrate improved referring accuracy and reduced hallucination, matching or exceeding supervised methods on various benchmarks.

Test-Time Computing for Referring Multimodal LLMs: ControlMLLM++

Introduction and Motivation

This paper presents ControlMLLM++, a test-time adaptation framework for Multimodal LLMs (MLLMs) that injects learnable visual prompts to enable fine-grained region-based visual reasoning in frozen, pre-trained MLLMs, obviating the need for further training or fine-tuning. The approach is fundamentally motivated by the observation that current MLLMs predominantly exploit coarse image-level alignments, failing to robustly support explicit region-level grounding—even when the text prompt implicitly refers to subregions. Prior work in this area introduces explicit referring capabilities via additional supervised training on region-text pairs, but such solutions are computationally intensive and often exhibit poor out-of-distribution generalization.

ControlMLLM++ addresses these limitations by leveraging and optimally perturbing cross-modal attention maps at test time, thus steering the model's focus toward user-specified regions indicated via diverse visual prompt types (e.g., box, mask, scribble, point). This yields task-adaptive region grounding while retaining the language understanding prowess and zero-shot generalizability of the base MLLM.

Figure 1: Comparison between training-based and test-time computing approaches. The test-time computing method allows prompt-based adaptation to new domains.

Technical Framework

Attention Map Analysis and Manipulation

A central hypothesis in the work is that semantic correspondences between language tokens and image regions are encoded in the cross-attention matrices of MLLMs. The authors validate that these attention maps, particularly those focused on high-impact text tokens (such as the answer-start token), capture localization cues across network layers.

Figure 2: Visualization of attention maps in MLLM layers. The top line captures the association between the prompt token “hat” and visual tokens; the bottom line pools attention across context tokens.

To manipulate model focus, the method injects learnable latent variables into the visual token streams. Several strategies for manipulating attention are compared, including direct coefficient-based adjustments and iterative latent variable optimization. Direct manipulation can distort the balance between prompt semantics and visual evidence, so the approach ultimately optimizes the visual tokens indirectly via a mask-based energy function, iteratively guiding the cross-attention distributions toward the relevant region.

Figure 3: Comparison of attention map manipulation methods, including coefficient-based and latent variable optimization approaches.

Figure 4: ControlMLLM framework overview. The framework uses a visual prompt-derived mask and an energy objective computed over attention maps. Latent variable optimization is performed iteratively at inference.

Latent Variable Optimization via Energy Functions

For region-based prompts (boxes, masks), the energy function penalizes misalignment between the pooled attention distribution and the mask, and gradient descent on the latent visual variable sharpens the model’s internal focus. For point or scribble inputs, a soft Gaussian mask is constructed for smooth spatial guidance.

Optimization is conducted solely at the zero-th decoding step and iterated for a small number of updates, preserving the MLLM’s generative fluency. The method employs stabilization heuristics such as Early Stopping and Exponential Moving Average, and further improves with an Adam-based optimizer and informative layer/text-token selection.

Enhanced Optimization: Optim++ and PromptDebias

Empirical analysis reveals that attention losses are concentrated in certain decoder layers (mid-depth), and that the answer-start token typically dominates text-to-visual alignment during output generation.

Figure 5: Contribution of text tokens to attention maps, focused on the answer-start token.

Figure 6: Decoder layer-wise loss distribution, with attention concentrated in middle layers.

Optim++ focuses optimization on the most informative layers and answer-start token, removing noise and accelerating convergence. PromptDebias is introduced to address persistent language bias and multimodal hallucination; it enforces a contrastive logit manipulation that subtracts visually agnostic decodings, thereby compelling the output distribution to be affected by the visual prompt and not merely inherent linguistic priors.

Figure 7: (a) ControlMLLM focuses on the correct region but outputs the same answer as the baseline due to language bias; (b) Different prompt phrasings lead to different outcomes, evidencing bias.

Empirical Validation

Referring Capability and Interpretability

ControlMLLM++ provides strong control over region-level referring across all common visual prompt formats. Qualitative analysis demonstrates sharper and more interpretable alignment between the referred regions and generated descriptions, with a reduction in hallucinated content.

Figure 8: Qualitative examples of all supported visual prompt types, with improvements in controllability and hallucination mitigation.

On out-of-domain OCR tasks, the approach enables models with no explicit grounding supervision to accurately identify local text content—surpassing specialized supervised baselines.

Figure 9: Comparison of OCR localization across models; ControlMLLM++ variant yields accurate regional text identification.

Quantitative Performance

ControlMLLM++ achieves:

• ROC (Referring Object Classification) accuracy on par with or exceeding supervised methods on in-domain (LVIS) and out-of-domain (COCO-Text) test sets, e.g., 71.19% (box) surpassing GPT4-ROI and nearly matching state-of-the-art Ferret.
• Substantial improvements over vanilla LLaVA architectures (e.g., from ~54.7% to 71.19% in ROC with box).
• Enhancement of even advanced, natively referring-capable models such as Qwen2.5-VL, especially on domain shift evaluations.

In referring description tasks (RefCOCOg, Screenshot) measured by BLEU, CIDEr, and SPICE, ControlMLLM++ delivers higher performance and robustness to distribution shift, notably improving generalization where conventional models substantially degrade.

Ablation and Efficiency

Ablation studies show that each component (Adam optimizer, informed attention selection, PromptDebias) incrementally contributes to alignment accuracy. Model scaling alone (e.g., 7B vs. 13B) does not yield similar returns as test-time control, validating the criticality of the computational framework over brute-force parameter scaling.

Figure 10: Ablation study on optimization stability and the effect of language bias mitigation weight.

Inference cost increases moderately due to gradient-based optimization but remains practical; PromptDebias introduces extra decoding for contrastive logit combination but may be amortized by efficient software frameworks.

Limitations

The method requires access to model gradients and internal representations, limiting applicability to open-source MLLMs. Extension to multi-region prompts is nontrivial due to potential gradient conflicts. Additional inference overhead is present but manageable, particularly with engineering optimizations.

Theoretical and Practical Implications

By leveraging test-time latent variable optimization, ControlMLLM++ establishes a new computational paradigm for controlling grounding and referential precision in MLLMs. It decouples the acquisition of fine-grained spatial alignment from high-cost dataset collection and supervised training. Practically, this enables plug-in referential reasoning for a wide variety of MLLM architectures—even those released without spatial supervision.

Theoretically, the results underscore the richness of cross-attention maps as semantic bottlenecks for language-image interaction, and suggest that informed test-time manipulation is a viable path for efficient adaptation and debiasing. Integrating additional priors or optimization strategies, scaling to multiple prompts, or extending to video-oriented or temporal region referencing are promising directions.

Conclusion

ControlMLLM++ offers a robust, test-time solution for explicit referring in Multimodal LLMs via efficient, targeted optimization of internal attention dynamics, rendering explicit retraining with region annotations unnecessary. This framework facilitates generalization, mitigates multimodal hallucination, and provides practical interpretability, representing a significant step toward more adaptable and controllable MLLM deployment.

(2602.19505)