EDC-118K: Annotated Image Captioning Corpus

• EDC-118K is a large-scale image captioning dataset that augments COCO images with synthetic captions and detailed object and relation metadata.
• It integrates fine-grained visual attributes and spatial grounding using specialized models to enhance multimodal perception.
• The dataset’s comprehensive annotations support reproducible research and the training of large multimodal models for advanced visual reasoning.

The EDC-118K dataset is a large-scale, richly annotated image captioning corpus designed to advance multimodal perception through enhanced descriptive captions. It systematically augments conventional captions with structured visual attributes—including fine-grained object details and explicit spatial/relational grounding—by leveraging a suite of specialized visual recognition models. EDC-118K is constructed atop the entire Microsoft COCO train2017 set, comprising approximately 118,000 images, and provides detailed object- and relation-level metadata suitable for complex visual reasoning and large multimodality model (LMM) training (Sun et al., 2024).

1. Dataset Composition

EDC-118K draws its images directly from Microsoft COCO train2017, incorporating all 118,287 images (rounded to 118,000). The dataset is treated as one pooled set; no train/validation/test partitions are defined within EDC-118K itself. Each image is annotated with a synthetic caption that merges the original COCO caption with extensive visual attributes, and paired with a structured metadata record containing object- and relation-level information.

Dataset Structure

Aspect	Details	Source/Notes
Total images	118,000	Rounded from 118,287 COCO
Source	Microsoft COCO train2017	http://cocodataset.org
Splits	None (single pooled set)
Caption format	Augmented: reference + attributes	Object, relation level
File format	JSON-lines: 1 record/image	See Section 5

2. Annotation Attributes and Definitions

Annotations in EDC-118K are divided into object-level and relation-level attributes, derived via off-the-shelf visual specialists not originally trained for captioning.

Object-Level Attributes

• Category (coarse): Detected using GroupDetr (in-domain) or LaMI-DETR (open-world).
• Color, Texture: Extracted from detection model features with rule-based postprocessing.
• Depth: Per-object scalar via Depth Anything V2, localized using bounding boxes and the global depth map.
• Emotion: Facial expression labels (e.g., happy, sad) assigned by Savchenko et al.'s recognizer to objects containing faces.
• OCR Text: Embedded text within object crops via Baidu OCR General API.
• Fine-grained classes: Where appropriate, specialist models override coarse categories:
  • Animals/plants: BioClip (Stevens et al.)
  • Aircraft: MMALNet
  • Logos/landmarks/celebrities: Baidu Image Recognition API
  • Food: PreNet (Min et al.)

Relation-Level Attributes

• Human-Object Interaction (HOI): List of triplets ⟨human, verb, object⟩ from RLIPv2 (Yuan et al.).
• 2D Absolute Location: Normalized bounding-box centroids mapped to discrete quadrants (e.g., "bottom-right").
• 2D Relative Location: Spatial relation between the object pair with greatest normalized center-to-center distance (e.g., "to the left of"). Only one dominant relation retained per image.
• 3D Relative Location: Depth-based ordering (e.g., "vase behind flowers") between the dominant object pair.

3. Data Collection and Annotation Protocol

The annotation protocol is a staged pipeline designed to maximize attribute coverage and minimize annotation errors:

1. Detection: Object detection to obtain bounding boxes and coarse categories.
2. Specialist Inference: Each crop is processed for depth estimation, OCR, emotion detection, and fine-grained classification. HOI triplets extracted for the entire image.
3. Object-level Prompt Construction: Merges the original COCO caption with extracted object-level attributes (enforcing overrides where specialist predictions differ).
   • Fine-grained labels replace coarse names if present; conflicts default to specialist prediction.
   • OCR text omitted if already reflected in the reference caption.
4. Relation-level Prompt Synthesis: All HOI and 2D absolute locations are retained. 2D and 3D relative relations pruned to one dominant pair each to avoid combinatorial explosion.
5. Quality and Consistency Controls: Single-turn, structured LLM prompts prevent drift and hallucination. Open-source recognition models were replaced by Baidu APIs where found insufficiently accurate.

4. Statistical Summaries

Caption statistics and relational analysis highlight the increased granularity and coverage relative to both human and LMM-generated baselines.

• Caption Lengths (tokens, Vicuna-v1.5 tokenizer):
  • COCO human: 14.7
  • InternVL2-26B: 105.8
  • LLaVA-NeXT-34B: 227.7
  • EDC-118K: 217.7
• Lexical Composition: EDC-118K captions exceed other sets in the average counts of nouns, verbs, adjectives, adverbs, and numerals per caption.
• Spatial Terms: EDC captions contain spatial expressions (e.g., "left side", "behind") at much higher frequency than LMM-generated baselines.
• Relation Counts: Each image has one dominant 2D relative and one 3D relative relation, plus all detected HOI and all 2D absolute locations (pruned only for redundancy).

No explicit probability tables or further LaTeX formulas beyond these averages are provided in the original manuscript.

5. File Formats and Technical Access

EDC-118K metadata for each image is released as a single JSON-lines file:

{ "image_id": …, "filename": …, "edc_caption": "…", "objects": [ { "id": 0, "bbox": [x, y, w, h], "category": "dog", "fine_category": "golden retriever", "color": …, "texture": …, "depth": …, "ocr_text": "…", "emotion": "…" }, … ], "relations": [ { "type": "HOI", "subject_id": 3, "verb": "holding", "object_id": 7 }, { "type": "2D_abs_loc", "object_id": 2, "location": "bottom-right" }, { "type": "2D_rel_loc", "subject_id": 2, "relation": "to the left of", "object_id": 5 }, { "type": "3D_rel_loc", … } ] }

Researchers can acquire accompanying COCO images from http://cocodataset.org; the EDC-118K JSON-lines file is distributed via the project GitHub repository upon public release. Full reproduction requires executing the EDC pipeline, which encompasses the detection, depth, OCR, emotion, fine-grained, HOI modules, followed by two prompt-driven merging stages.

6. Reproducibility and Use Considerations

The construction protocol enables reproduction or extension for aligned vision-language research. Researchers seeking to replicate EDC-118K should begin with COCO train2017 images, execute the multi-stage EDC annotation pipeline, and integrate the outputs into the merged caption plus attribute format. The resulting dataset supports evaluation and pretraining of LMMs and visual understanding systems requiring attribute-rich and spatially grounded natural language descriptions (Sun et al., 2024).

A plausible implication is that the structured, multi-attribute annotation design can facilitate new benchmarks for fine-grained visual reasoning as well as serve as a pretraining resource for tasks needing high-fidelity scene grounding. The requirement to use specialist, sometimes proprietary, API services for OCR and entity recognition suggests reproducibility will depend on comparable access or equivalently accurate alternatives.