LLM-Grounder: Modular Multimodal Reasoning

• LLM-Grounder is a paradigm that employs LLMs as central agents to decompose natural language queries and coordinate proposals across diverse modalities.
• It integrates language query decomposition, modality-specific proposal detection, and reasoning-based selection to enhance compositional grounding.
• Empirical results show significant accuracy gains in 2D, 3D, video, and web environments, making it a practical and scalable approach.

LLM-Grounder refers to a class of systems and architectural paradigms that leverage LLMs as central agents or reasoning modules for grounding tasks in perception and information environments. In this context, "grounding" denotes the process of mapping free-form natural language queries to explicit, localized entities across modalities such as 2D images, 3D scenes, video, structured documents, or dialog environments. The LLM-Grounder paradigm characteristically replaces or wraps traditional metric-based or trainable vision-language pipelines with a reasoning-centric approach: the LLM (potentially augmented by modular tool use, symbolic interfaces, or downstream neural models) orchestrates query decomposition, modality-specific proposal, and final decision-making. State-of-the-art systems demonstrate that LLM-grounded architectures deliver substantial accuracy and compositionality gains, especially under open-vocabulary, zero-shot, or complex referential scenarios.

1. Core LLM-Grounder Pipeline Concepts and Variants

The canonical LLM-Grounder architecture integrates three stages: (1) a language query decomposition agent based on an LLM, (2) a set of proposal or detection back-ends (e.g., visual, structural, or symbolic tools), and (3) an LLM-enabled or symbolic selection mechanism that fuses outputs and applies reasoning constraints. Early and recent instantiations in 2D, 3D, and web environments have adopted this pattern, often with zero-shot or minimal fine-tuning requirements.

For 2D visual grounding, LLM-Optic (Zhao et al., 2024) exemplifies a high-accuracy LLM-Grounder pipeline by introducing a Text Grounder that reduces compositional queries to concise targets, a detector that produces marked candidates, and a Visual Grounder (multimodal LLM) that reasons over candidate marks and the original query. In 3D, LLM-Grounder (Yang et al., 2023) and evolvable symbolic variants (Mi et al., 3 Feb 2025) place an LLM at the center: first decomposing queries into targets, landmarks, and spatial relations, then interfacing with CLIP-based 3D proposal tools (e.g., OpenScene or LERF), and finally using the LLM to aggregate geometric and commonsense cues for disambiguation.

The paradigm extends naturally to web automation domains, as in Prune4Web (Zhang et al., 26 Nov 2025), where an LLM defines semantic filters for DOM element selection, and subsequent grounding is performed over a reduced, programmatically scored candidate set. In video and temporal localization, VideoMind (Liu et al., 17 Mar 2025) employs an agentic workflow dividing planning, temporal grounding, verification, and answering into specialized LoRA-adapted LLM modules.

2. Algorithmic and Mathematical Formulations

LLM-Grounders generally employ deterministic or instruction-prompted output interfaces to enforce logic constraints or candidate selection. For instance, LLM-Optic’s Text Grounder calls a single-turn prompt to GPT-3.5, yielding a subject via JSON extraction; candidate boxes are then numerically marked and presented to a multimodal LLM for discrete selection (Zhao et al., 2024).

Mathematically, grounding accuracy is typically measured using Intersection-over-Union (IoU) and Accuracy@τ:

• $$\mathrm{IoU}(b, b^*) = \frac{\mathrm{area}(b \cap b^*)}{\mathrm{area}(b \cup b^*)}$$
• $$\mathrm{mIoU} = \frac{1}{M}\sum_{k=1}^M \mathrm{IoU}(b_k, b_k^*)$$
• $$\mathrm{Accuracy@τ} = \mathrm{Fraction\; of\; queries\; with\;} \mathrm{IoU} \geq \tau$$

For web agents (Zhang et al., 26 Nov 2025), the LLM produces a dictionary of keyword–weight pairs, plugged into a scoring function:

$$s(e) = \sum_{(k,w_k)\in W} w_k \cdot \sum_{(text,typ)\in\text{attrs}(e)} \alpha(k, text) \cdot \beta(typ)$$

Here, $\alpha$ and $\beta$ weight match-quality and attribute priority, respectively.

In symbolic 3D grounding (Mi et al., 3 Feb 2025), the LLM generates relation-encoder snippets, which compute scores for geometric/spatial predicates, e.g., for “near”:

$$f^{\rm near}_{ij} = \exp\left(-\frac{d_{ij}}{s_{\rm avg}+\varepsilon}\right)$$

3. Empirical Performance and Evaluation

LLM-Grounders consistently deliver state-of-the-art or strong zero-shot results across multiple modalities.

• On RefCOCOg Val, LLM-Optic lifts [email protected] from 0.505 (Grounding DINO) to 0.725 (+22 ppt), with similar mIoU and [email protected] gains (Zhao et al., 2024).
• Prune4Web achieves 88.28% grounding accuracy (vs 46.8% for a baseline LLM-only model) by combining programmatic DOM pruning with LLM-based candidate classification (Zhang et al., 26 Nov 2025).
• On ScanRefer (3D), LLM-Grounder raises [email protected] for OpenScene from 13.0% to 17.1% and [email protected] from 5.1% to 5.3% (Yang et al., 2023). The Evolvable Symbolic Visual Grounder (EaSe) attains 52.9% on Nr3D, at an order-of-magnitude lower inference cost than agentic alternatives (Mi et al., 3 Feb 2025).
• In video, VideoMind’s Grounder module achieves [email protected] = 47.2 and mIoU = 42.0 on Charades-STA, matched only by full fine-tuning at far higher parameter cost (Liu et al., 17 Mar 2025).

These improvements are most significant for compositional, multi-object, or spatially complex queries, confirming the LLM’s reasoning advantage over prior closed-set detectors.

4. LLM-Grounder Design Across Modalities

LLM-Grounder blueprints have been adapted to diverse environments:

• Images: LLMs interpret complex textual expressions, aggregate or rerank detector outputs, and handle spatial/semantic ambiguity (Zhao et al., 2024).
• 3D Scenes: LLMs decompose language into symbolic sub-queries, invoke CLIP-based or code-generated geometric relation evaluators, and select via spatial and commonsense reasoning (Yang et al., 2023, Mi et al., 3 Feb 2025).
• Video Segments: Specialized Grounder modules predict start/end frames by decoding timestamp tokens, optionally under LoRA adaptation for efficiency, with role-specific prompt routing (Liu et al., 17 Mar 2025).
• Web Agents: LLMs express pruning logic as Python code to filter DOM trees, reducing candidate space prior to LLM classification (Zhang et al., 26 Nov 2025).
• KGQA and Event Extraction: LLM-Grounders recast free-form reasoning as bounded discriminative or constrained decoding problems, thus lowering hallucination rates and enforcing ontology or graph consistency (Xu et al., 2024, Parekh et al., 5 Jun 2025).

Common to all variants is a tight LLM-in-loop architecture for both decomposition and selection, often interfacing with small, specialized, or frozen modalities without retraining.

5. Ablations, Limitations, and Open Questions

Ablation studies show that the particular choice of LLM is less important for syntactic extraction (subject/target identification) but critical for semantic or multimodal selection. For example, GPT-3.5 and open-source models ≥7B perform comparably for subject extraction, but Visual Grounder selection benefits from high-capacity architectures (GPT-4V outperforms LLaVA-1.6 on mIoU by up to 17 points) (Zhao et al., 2024).

Limitations persist:

• Cost and latency from LLM/LMM API calls, particularly in closed-loop or agentic settings (Yang et al., 2023, Zhao et al., 2024).
• Reliance on prompt quality for ambiguous or under-specified queries (e.g., “black shirt, light blue jeans”) (Zhao et al., 2024).
• Rare but present failure to parse candidate markers or misinterpret visual–textual bridges.
• Lack of end-to-end joint optimization in most current modular systems.
• For complex 3D or visual grounding, spatial granularity of proposals is limited by the default back-end (often CLIP-based clusterers), not by LLM logic.

6. Implications for Generalization and Future Directions

LLM-Grounders point toward a paradigm shift: from fully end-to-end vision-LLMs to modular, interpretable, and tool-using reasoning architectures, where the LLM orchestrates and adjudicates over a set of proposals. This approach enables universal, zero-shot grounding, extensible to arbitrary objects and queries without task-specific re-training.

Anticipated directions include:

• Local deployment of open-source LLMs/LMMs to mitigate cost/latency (Zhao et al., 2024, Mi et al., 3 Feb 2025).
• Adaptive, multi-turn or interactive prompt engineering to clarify ambiguous intent.
• Joint or reinforcement learning over LLM-grounder modules.
• Integration of finer-grained spatial, attribute, or mask-level outputs using advances in decoupled and fusion-driven architectures (Jisheng et al., 28 Jun 2025).
• Application in dialogue grounding, decision-making (via hierarchical RL) (Hu et al., 26 May 2025), and robust presupposition handling in conversational settings (Lachenmaier et al., 10 Jun 2025).

The modular LLM-Grounder architecture thus provides a blueprint for next-generation multimodal understanding, combining the compositional power of LLMs with the precision of domain-specific proposal engines.