LLaVA-Grounding Models

• LLaVA-Grounding is a framework that extends large multimodal models with fine-grained visual grounding by unifying visual chat and detailed region localization.
• It integrates vision and language encoders with segmentation and detection modules while leveraging a rigorously annotated dialogue dataset for explicit grounding.
• The approach achieves state-of-the-art performance on grounding benchmarks, enhancing both pixel-level and box-level accuracy in image and video contexts.

LLaVA-Grounding (LLaVA-G) refers to a series of architectural extensions and dataset/methodological innovations that equip Large Multimodal Models (LMMs)—notably LLaVA and LLaVA-1.5—with fine-grained visual grounding capabilities. The resulting models unify detailed referential expression comprehension, pixel- or box-level localization, and visual chat, establishing a framework for grounded conversational visual intelligence. The LLaVA-G paradigm is characterized by modular integration of vision and language encoders with segmentation or detection heads, a rigorously constructed supervised dataset for grounded dialogues, and end-to-end differentiable connections between conversational outputs and visual region selection. The family includes both image- and video-centric variants, as well as grounded instruction datasets and specialized evaluation benchmarks (Zhang et al., 2023, Kang et al., 11 Aug 2025, Munasinghe et al., 2023).

1. Grounded Dataset Construction and Annotation Paradigms

Central to LLaVA-Grounding is the construction of the Grounded Visual Chat (GVC) dataset (Zhang et al., 2023). The data pipeline integrates natural-image instances (MS COCO 2017) and multi-turn, GPT-4–curated visual chat responses. For each image-dialogue pair, all noun phrases in the language responses are matched with ground-truth (GT) box and segmentation annotations via GPT-4, producing entity-phrase alignments. Special tokens $\langle g_s\rangle$ ... $\langle g_e\rangle$ are used for marking grounded phrases, and $\langle seg\rangle$ tokens indicate the position in the text stream for which grounding predictions (boxes/masks) are needed. The pipeline supports various referencing paradigms, including direct region input via clicks, boxes, or marks. The final dataset encompasses $\sim$150K grounded visual-sentence pairs, with additional referring variants (GVC-R) extending coverage to single-object and visual-prompted settings.

The GVC annotation design ensures each linguistic span referenced in dialogue is associated with precise visual regions. This enables training with explicit supervision for both chat and grounding, addressing the historic lack of datasets combining these modalities. Other LLaVA-G implementations extract referring/grounding examples from existing datasets (e.g., RefCOCO/+/g) and deduplicate conversational turns to maximize data efficiency and minimize label leakage (Kang et al., 11 Aug 2025).

2. Architectural Framework and Design Variants

All LLaVA-Grounding models build upon the LLaVA vision-language architecture, typically consisting of a CLIP-ViT vision encoder connected to a Vicuna LLM via a linear or MLP connector. The grounding extension introduces two primary architectural innovations (Zhang et al., 2023):

• Visual Interaction (Prompt) Encoder: Processes the image along with an optional visual prompt (click, box, scribble, mark), deriving prompt-specific embeddings via a frozen segmentation model (e.g., Semantic-SAM), then projecting into the LLM token embedding space. At inferred $\langle obj\rangle$ token locations, the textual embedding is replaced by the prompt embedding.
• Grounding Model: For every $\langle seg\rangle$ token in the language output, the LLM's last-layer hidden state is projected into the query space of a frozen OpenSeeD (or similar) segmentation/detection model. The grounding model then predicts the bounding box ($\mathbf{B}$) and/or pixel-level mask ($\mathbf{M}$) corresponding to each grounded span. These components are trained to align with GT grounding, providing pixel- or box-level localization for arbitrary free-form text spans.

Alternative paradigms explored (Kang et al., 11 Aug 2025) involve autoregressive box value prediction as explicit integer tokens (0–100, normalized to [0,1]), either as a sequence in the output stream or via special hidden-state transformations. Integer tokenization was empirically favored, as it aligns with Vicuna's native autoregressive prior and yields stronger referential accuracy.

In video settings, LLaVA-Grounding employs a modular system comprising (a) frame-level CLIP-ViT feature extraction; (b) spatio-temporal pooling and an MLP projector; (c) audio transcription integration (using WhisperX+Whisper-AT and VAD for speaker-centric context enrichment); (d) phrase extraction from dialogue for each referent; and (e) a plug-and-play inference pipeline of noun-phrase extraction followed by GroundingDINO + SAM + DEVA tracker for spatiotemporal region localization (Munasinghe et al., 2023).

3. Training Objectives and Procedures

LLaVA-Grounding employs hybrid training regimes, optimizing both linguistic and grounding-specific targets. The objective is typically a weighted sum:

$$\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{lang}} + \lambda_{\mathrm{box}} \mathcal{L}_{\mathrm{box}} + \lambda_{\mathrm{mask}} \mathcal{L}_{\mathrm{mask}} + \lambda_{\mathrm{match}} \mathcal{L}_{\mathrm{match}}$$

• $\mathcal{L}_{\mathrm{lang}}$: Cross-entropy over next-token prediction in language outputs.
• $\mathcal{L}_{\mathrm{box}}$: Smooth-$L_1$ regression of predicted vs. GT box coordinates.
• $\mathcal{L}_{\mathrm{mask}}$: Pixelwise binary cross-entropy over predicted vs. GT segmentation maps.
• $\mathcal{L}_{\mathrm{match}}$: Cross-entropy on the assignment between predicted queries and GT entity instances.

Training follows a curriculum: (1) vision-language and grounding pre-alignment, (2) instruction tuning on combined chat and grounding data, (3) targeted adaptation for visual prompting. Pretrained CLIP and segmentation heads are frozen, with only connection and projector layers updated alongside Vicuna. Hyperparameters are typically inherited from official LLaVA or OpenSeeD codebases: AdamW, $\mathrm{lr}=2\mathrm{e}{-5}$ (LLM), batch size 64, 4 epochs found optimal empirically for classification-based box grounding (Kang et al., 11 Aug 2025, Zhang et al., 2023).

Notably, some variants (e.g., PG-Video-LLaVA) employ only the standard language modeling loss, with grounding performed exclusively at inference using frozen detectors/trackers, i.e., “zero-shot” grounding (Munasinghe et al., 2023).

4. Grounding Benchmarks, Metrics, and Results

LLaVA-Grounding introduces the Grounding-Bench benchmark, comprising 1,000 MS COCO val images with 7,000+ associated entities and multi-turn, GPT-4–evaluated chat sessions (Zhang et al., 2023). The evaluation protocol includes:

• Chat Quality: Human-aligned scoring of conversation, detailed description, and reasoning (LLaVA Bench style).
• Grounded Response Metrics: Recall ($R$), Precision ($P$), and $F_1$ for each mention-mask alignment:
  • $R = TP_{gt}/N_{gt}$ (fraction of GT entities correctly mentioned/grounded)
  • $P = TP_{pred}/N_{pred}$ (fraction of predicted groundings that correctly match GT)
  • $F_1 = 2PR/(P+R)$

True positives require both IoU $>$ 0.5 and semantic correctness (evaluated via GPT-4).

Performance on Grounding-Bench:

• LLaVA-G-7B: Recall 28.6%/36.3%, Precision 52.7%/53.4%, $F_1$ 37.1%/43.2%, outperforming other 7B-parameter LMMs by $\sim$5–10$F_1$ (Zhang et al., 2023).

On classic referring/grounding datasets:

• RefCOCO ([email protected]): LLaVA-G 89.16%, mIoU 77.13%
• RefCOCO+ ([email protected]): LLaVA-G 81.68%, mIoU 68.79%
• RefCOCOg ([email protected]): LLaVA-G 84.82%, mIoU 71.54% Absolute gains over LLaVA-1.5: +5.6 / +6.9 / +7.0 p.p. on RefCOCO/+/g (Kang et al., 11 Aug 2025).

Prompt-based video benchmarks are also introduced:

• VidSTG: LLaVA-G achieves mean IoU 34.2% (7B), 35.1% (13B)
• HC-STVG: mean IoU 28.3% (7B), 27.3% (13B) These outperform GroundingDINO and previous LMM video baselines (Munasinghe et al., 2023).

5. Ablation Studies and Optimal Design Choices

Comprehensive ablation on data and model design yields key insights (Kang et al., 11 Aug 2025):

• Box prediction as integer tokens ([0,100] per normalized coordinate) and cross-entropy loss yield superior performance, aligning with LLM priors.
• Limiting dialogues to $\leq3$ turns and deduplicating Q-&-A pairs with redundant boxes enhances generalization (+4–6 p.p.).
• Expanding the proportion of visual grounding data (VG) relative to visual question answering (VQA) is more effective than mixed training (“scaled-VG” $>$ VG+VQA $>$ VG only).
• Four training epochs maximize grounding performance.
• Detaching grounding gradients marginally improves chat but substantially degrades $F_1$.

These design principles collectively define the LLaVA-G training "recipe."

6. Limitations, Failure Modes, and Future Directions

LLaVA-Grounding architectures demonstrate strong modularity, permitting architectural innovation and open-vocabulary, multi-modal reference. Limitations include:

• Semantic scope is bounded by the available GT instance classes (MS COCO, Visual Genome).
• Error modes include noun-phrase extraction failures, object detection/segmentation errors, and, in video, identity swap during tracking.
• Video grounding is inference-only, with no fine-tuning of grounding heads, limiting the capacity to address video-specific ambiguities (Munasinghe et al., 2023).

Anticipated future directions involve expanding grounding to 3D scenes, open-vocabulary semantics, and real-time video; integrating temporal attention for motion disambiguation; and evolving from modular zero-shot ensembles to jointly-trained transformers capable of unified pixel-level predictions. Applications span interactive image editing, robust visual chat, referential instruction for robotics, and open-set object grounding in conversational agents (Zhang et al., 2023).