---
title: 'GranD: Pixel-Level Vision-Language Dataset'
url: https://www.emergentmind.com/topics/grounding-anything-dataset-grand
type: topic
---
# GranD: Pixel-Level Vision-Language Dataset
The Grounding-anything Dataset (GranD) is a large-scale, richly annotated vision–language dataset designed to support dense, pixel-level grounding of natural language in natural images. Developed to enable and benchmark models for visually grounded conversation, segmentation, captioning, and other fine-grained multimodal tasks, GranD leverages automated pipelines based on state-of-the-art vision and language models to provide region-level annotations, dense captions, relationship graphs, and contextual information for millions of images. Its scale, granularity, and automated construction distinguish it as a foundational resource for research at the intersection of computer vision and language modeling [2311.03356].
## 1. Dataset Construction Pipeline
GranD is constructed by annotating 11 million images (sourced from the SA-1B "Segment Anything" dataset) through a hierarchical, four-stage pipeline. Each stage employs specialized off-the-shelf models for distinct annotation tasks and uses the output of prior stages as input.
1. **Level 1: Object Localization & Attributes**
- Multiple object detectors (Co-DETR, EVA-02, OWL-ViT, POMP) are run on each image.
- Class-agnostic non-maximum suppression merges output bounding boxes, retaining boxes only if at least two detectors agree ($\mathrm{IoU} > 0.5$).
- For each region $r_i$, category and attribute labels ($c_i$, $a_i$) are generated using GRiT or GPT4RoI, and depth $d_i$ is estimated with MiDAS.
- Each $r_i$ is matched to a panoptic instance mask $M_i$ from the SA-1B pool using $\mathrm{IoU}$ matching.
- Output: List of detected regions with associated masks, categories, attributes, and depth.
2. **Level 2: Relationships & Landmarks**
- Scene descriptions are generated using BLIP-2 or LLaVA, producing a set $S = \{s_1, ..., s_k\}$ per image.
- Noun phrases $\phi_j$ are extracted from $S$ via spaCy, and grounded to regions using MDETR.
- Relationship triplets $(\phi_{j1}, \mathrm{rel}, \phi_{j2})$ are assembled.
- LLaVA assigns one of four main image landmark categories (indoor, outdoor, transportation, sports) and associated subcategories.
3. **Level 3: Scene Graph & Dense Captioning**
- Hierarchical scene graphs organize objects and relations by depth (foreground $\to$ midground $\to$ background).
- Dense region-level captions are generated by prompting Vicuna-13B with the scene graph, requiring at least three alternating text+mask references per caption.
- Automatic verification: object mentions in captions must align with the scene graph, with prompts iteratively modified until all objects are referenced.
4. **Level 4: Extra Context**
- Vicuna-13B is prompted with the scene graph and captions to produce a "beyond-the-image" context paragraph per image (e.g., historical facts, usage, or likely events).
No human-in-the-loop is required beyond prompt engineering; all annotation steps use deterministic rules and model outputs [2311.03356].
## 2. Coverage, Statistics, and Data Structure
GranD provides annotations at a globally unmatched scale and detail:
| Statistic | Value | Notes |
|-----------------------------------|-------------------|---------------------------------------|
| Images | 11,000,000 | Sourced from SA-1B pool |
| Regions (maskable instances) | 810,000,000 | Pixel-level, each with mask + label |
| Unique concepts | 7,500,000 | Distinct object labels |
| Referring expressions | 84,000,000 | Short queries for instance grounding |
| Short grounded captions | 22,000,000 | Region-level |
| Dense grounded captions | 11,000,000 | Per-image, interleaved with masks |
| Image-caption-mask triads (GCG subset) | 214,000 | For GCG fine-tuning/test/val |
Concept density per image is high, with a mean of approximately 73.6 unique concepts per image. Empirically, the region area distribution is long-tailed: 60% of regions occupy less than 2% of image area, 20% occupy 2–10%, and the remainder exceed 10%. Concept density (regions per 1,000 pixels) averages 0.015.
The taxonomy follows four main landmark categories (Indoor, Outdoor, Transportation, Sports) and their respective subcategories (e.g., Outdoor: Urban, Rural, Natural landscape).
GranD is released under a permissive academic license. Its directory structure is organized as:
- `images/`: original .jpg files
- `masks/`: per-region binary masks (run-length encoded .png)
- `annotations/`: one .json annotation file per image
Annotation schemas aggregate per-object (ID, category, attributes, depth, mask), relationships, region-level dense captions (with phrase–mask links), and image-level context.
## 3. Annotation Schema and Example
The annotation schema for a single image is structured as follows:
```json
{
"image_id": 12345,
"width": 1920,
"height": 1080,
"landmark": { "main": "Outdoor", "sub": "Urban landscape" },
"objects": [
{
"region_id": 0,
"category": "bicycle",
"attributes": ["red", "metal"],
"depth": 3.41,
"bbox": [x, y, w, h],
"mask_path": "masks/12345_0.png"
},
...
],
"relationships": [["bicycle", "next to", "person"], ...],
"dense_captions": [
{
"caption": "A man riding
a red bicycle
…",
"phrase_spans": [[[0,1],"A man riding"], [[1,2],"a red bicycle"], ...],
"phrase_masks": [0,1, ...]
}
],
"extra_context": "This bicycle appears in a Dutch city known for its cycling culture…"
}
```
Phrase-to-mask associations enable dense, explicit grounding of language to precise pixel regions. Region relations, landmark tags, and scene-level context provide further multimodal structure.
## 4. Downstream Tasks and Evaluation
GranD supports four pretraining tasks for the Grounded LMM (GLaMM) model [2311.03356]:
1. **Referring-Expression Segmentation:** Given a phrase $\phi$, segment the corresponding region in the image.
2. **Region-Level Captioning:** Produce a caption for a queried bounding box or region.
3. **Image-Level Captioning:** Generate a detailed global image caption.
4. **Grounded Conversation Generation (GCG):** Generate a natural language response interleaved with segmentation masks—a dense caption string with each phrase constrained to a corresponding mask.
For GCG, evaluation splits are: validation 2,500 images, test 5,000 images. Metrics include:
- Caption quality: METEOR, CIDEr (n-gram matching)
- Mask quality: AP$_{50}$ (average precision @ $\mathrm{IoU} \geq 0.5$), mean IoU ($mIoU = (1/N) \sum_{i} |M_i \cap \hat{M}_i| / |M_i \cup \hat{M}_i|$)
- Mask recall with text match: Recall = correctly grounded phrases / total GT phrases (requires BERTScore $\geq 0.5$)
For referring segmentation and captioning tasks, standard benchmarks and metrics (IoU, AP, SPICE) are used, following Visual Genome, Flickr30K, and refCOCO variants.
## 5. Applications, Extensions, and Significance
GranD’s broad and dense annotation base supports a variety of vision–language tasks:
- **Referring Expression Segmentation:** Segment any instance referenced by a short query in-language.
- **Dense and Region Captioning:** Fine-grained image region annotation to enable use cases in retrieval, accessibility, and human-computer interaction.
- **Visually Grounded Conversation:** Multi-turn, object-referential dialogue where each response is tied to physical image regions.
- **Phrase Grounding:** Training and evaluating models to map arbitrary text phrases to segmentation masks.
- **Generative Vision Tasks:** Mask extraction with GLaMM enables conditional inpainting, editing, or diffusion modeling.
The dataset’s scale and annotation richness substantially increase the granularity and coverage available in region-level and pixel-level vision–language pretraining. By releasing 11 million richly annotated images with over 810 million masks, GranD provides a foundation for scalable research into grounded multimodal learning, embodied AI, and interactive image editing tasks [2311.03356].
## 6. Context and Future Developments
GranD addresses the scarcity of large-scale, pixel-precise grounding datasets necessary for robust vision–language model pretraining and benchmarking. The automated, model-centric pipeline enables ongoing expansion and adaptation as new detectors and captioning models become available.
A plausible implication is that this approach—using large, diverse synthetic and human-authored detection/caption outputs in an automated framework—may become central in scaling and evaluating future multimodal systems. Ongoing work may extend GranD with more diverse scene types, finer-grained relation annotations, and integration of real-world image sources to further enhance generalization and domain transfer.
Source: https://www.emergentmind.com/topics/grounding-anything-dataset-grand