Detailed Image Descriptions (DoI)
- DoI is a framework for converting visual content into rich, detailed text that captures fine-grained attributes, spatial relations, and contextual information.
- The field leverages methods ranging from controlled human annotation to fully automatic pipelines to enhance factual grounding and reduce hallucination.
- Evaluation benchmarks use scene graphs and cross-modal metrics to ensure descriptions are discriminative, accurately grounded, and suitable for diverse downstream applications.
Searching arXiv for papers on detailed image descriptions and related evaluation/datasets. arxiv_search.query({"search_query":"all:(\"detailed image description\" OR \"image textualization\" OR DOCCI OR ImageInWords OR Densely Captioned Images OR PoSh)","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"}) arxiv_search.query({"search_query":"all:(\"image textualization\" OR DOCCI OR \"ImageInWords\" OR \"Densely Captioned Images\" OR PoSh)","start":0,"max_results":10,"sort_by":"submittedDate","sort_order":"descending"}) Descriptions of Images (DoI) are textual representations of visual content used to support image understanding, image-to-text generation, text-to-image generation, retrieval, evaluation, and accessibility. In the recent vision-language literature, the term increasingly denotes a shift away from short, weakly aligned captions toward long, grounded, discriminative, and richly factual descriptions. This shift is motivated by two recurrent observations: web-scraped image-text pairs are abundant but noisy, and standard human-labeled datasets are often succinct and lack detail. Newer datasets and frameworks therefore aim to maximize visual-to-textual conversion, reduce hallucination, and capture fine-grained attributes, relations, spatial structure, and contextual information (Pi et al., 2024, Onoe et al., 2024, Garg et al., 2024).
1. From short captions to detailed descriptions
Early large-scale work on image description emphasized controlled crowdsourcing rather than exhaustive detail. A notable example is the collection of PASCAL-50S and ABSTRACT-50S through Amazon Mechanical Turk, where workers were shown a single image and instructed to describe what is “going on” in the image. The collection protocol explicitly emphasized transcribing observable scene content rather than imaginative elaboration, required grammatical sentences, and enforced strong worker filtering: annotators had to be located in the United States, have at least a 95% HIT approval rate, and at least 500 approved HITs. Each image received 50 descriptions from distinct workers, yielding 50,000 descriptions for PASCAL-50S and 25,000 for ABSTRACT-50S; the average sentence lengths were 8.8 and 10.59 words respectively (Vedantam et al., 2014).
Cross-linguistic work subsequently showed that image description is not merely a language-generation task, but also a cultural and epistemic one. A comparison of Dutch, English, and German descriptions over the same images found broad similarities in description practices, including frequent use of indefinite articles, the presence of negation in all three languages, and recurrent inferential or bias-informed content. At the same time, specificity varied with annotator familiarity: Dutch annotators could identify a “street organ” and the Tuileries Garden, while English and German annotators often produced more generic or erroneous labels; sports images showed analogous familiarity effects, such as Dutch and German annotators mislabeling American football as rugby, and Americans mislabeling rugby as soccer or football (Miltenburg et al., 2017).
More recent datasets formalize a transition from captioning to dense description. Densely Captioned Images (DCI) contains 7,805 natural images with human-annotated, mask-aligned descriptions averaging 1,111 words each, while DOCCI provides long human-annotated English descriptions for 14,847 images averaging 136 words. ImageInWords (IIW) pushes this further with 9,018 images and descriptions averaging 9.8 sentences and 217.2 tokens. Collectively, these resources reposition DoI as a high-recall, high-specificity representation problem rather than a short-summary task (Urbanek et al., 2023, Onoe et al., 2024, Garg et al., 2024).
| Resource | Scale | Description regime |
|---|---|---|
| PASCAL-50S / ABSTRACT-50S | 1,000 / 500 images | 50 crowdsourced descriptions per image |
| DCI | 7,805 natural images | Mask-aligned descriptions averaging 1,111 words |
| DOCCI | 14,847 annotated images | Long human-annotated English descriptions averaging 136 words |
| IIW | 9,018 images | Hyper-detailed descriptions averaging 217.2 tokens |
2. Annotation and curation paradigms
DOI resources differ sharply in how they construct textual detail. DOCCI adopts a deliberately discriminative human annotation protocol built around connected and contrasting images. Its annotators follow a three-stage procedure: extract key image aspects, combine them into a first-pass draft description, and then elaborate and refine for completeness and distinguishing detail. Instructions explicitly target objects, attributes, spatial relationships, counting, text rendering, world knowledge, scenes, camera views, and optical effects. The resulting descriptions are sufficiently discriminative that, given a description and five similar images, annotators selected the correct image 97.1% of the time, with Fleiss’ kappa of 0.98 (Onoe et al., 2024).
IIW introduces a seeded, sequential, human-in-the-loop pipeline. Object detection models first identify regions, and PaLI-3 5B generates initial object-level and image-level captions. Annotators then refine object labels, bounding boxes, and descriptions, after which they synthesize a holistic, flowing, hyper-detailed report. Rather than parallel independent annotation, IIW uses sequential augmentation: each annotator augments or corrects the previous description. This procedure is reported to reduce time by 30% and increase detail by 20% in token count, while also improving agreement and consistency. The framework releases not only image-level descriptions but also object-level annotations and enriched versions of other caption datasets (Garg et al., 2024).
Image Textualization (IT) represents a different trajectory: fully automatic detailed description generation. Its pipeline has three phases. Holistic Textualization uses MLLMs such as LLaVA or GPT-4V to create a reference description; Visual Detail Textualization detects hallucinations by extracting object phrases and verifying them with an open-set detector such as Grounding DINO, then enriches the factual content via GRiT dense captioning, SAM masks, and monocular depth estimation; Textualized Recaptioning uses an LLM to synthesize a final fluent description conditioned on the reference description, hallucination list, and grounded object information. The framework explicitly computes object depth and relative size as
where is the object mask and the depth map (Pi et al., 2024).
These annotation regimes imply distinct notions of “quality.” Human-only protocols emphasize discriminability and prose control; human-in-the-loop protocols emphasize coverage and correction of seed outputs; fully automatic systems emphasize scalability and factual grounding. A plausible implication is that DoI research is converging on hybrid pipelines in which linguistic fluency, localized perception, and explicit verification are treated as separate but composable subproblems.
3. Granularity, grounding, and structural representation
A central technical issue in DoI is the unit of alignment between image and text. DCI treats description as region-grounded and hierarchical rather than purely global. Its annotations are associated with explicit image regions via Segment Anything Model masks and organized in a mask-submask tree, enabling subcrop-caption matching rather than only image-level retrieval. Because CLIP-style models are often restricted to 77 text tokens, DCI also introduces sDCI, a summarized variant in which caption length is limited but remains more detailed than conventional captions; full-image descriptions average 1,282 CLIP tokens, while summarized captions average 49.21 tokens (Urbanek et al., 2023).
DOCCI emphasizes compositional density in whole-image prose. Its dataset analysis reports that 99.9% of descriptions mention spatial relations, 54.6% discuss counting, and 23.3% involve text rendering. The descriptions are not merely long; they are structured to disambiguate similar scenes through orientation, location, direction, relative position, and attribute binding. This makes DOCCI especially relevant for problems where omission and attachment errors dominate over gross object-recognition errors (Onoe et al., 2024).
IT extends grounding beyond region descriptions to explicit object tuples containing phrase, depth, and size, then uses those tuples in recaptioning. This design addresses a common failure mode of MLLM-only generation: plausible but unsupported elaboration. By separating hallucination detection from fact enrichment, IT treats detailed description as a grounded synthesis task rather than unrestricted generation (Pi et al., 2024).
DoI methods also extend beyond ordinary photographic scenes. Sat2Cap uses 6.1 million pairs of overhead and ground-level images to map satellite imagery into a textual space without text-labeled training data. A frozen CLIP image encoder produces ground-level embeddings, a trainable overhead encoder predicts corresponding embeddings from satellite images, and a dynamic encoder adds date-time and location information. The final embedding enables zero-shot mapping of arbitrary text queries and temporally varying concepts such as seasonal or time-dependent ground-level phenomena (Dhakal et al., 2023).
Sequential imagery introduces yet another structural regime. An end-to-end CNN-LSTM model for sequential images combines local-object attention with global semantic context modelling, using VGG16 global and local features, an MLP to initialize a shared semantic context, and paralleled LSTM decoders for sentence generation across an image sequence. On Microsoft’s DII dataset, it improves over a cnn-att-lstm baseline on BLEU, METEOR, and ROUGE, indicating that DoI can also denote coherent multi-image narrative description rather than only single-image captioning (Su et al., 2020).
4. Evaluation regimes and benchmark design
Evaluation has become a defining difficulty for DoI because long descriptions concentrate errors in attributes, relations, omissions, and attachment structure rather than simple object presence. PoSh addresses this by using scene graphs as structured rubrics for LLMs-as-a-Judge. For a description , the scene graph contains objects, attributes, and relations; an open-weight LLM then answers templated questions over scene-graph components in both generation-to-reference and reference-to-generation directions, producing precision-oriented mistake scores and recall-oriented omission scores. The associated DOCENT benchmark contains 1,750 public-domain artworks, expert-written references averaging 251 words and 161 scene-graph details, and both granular and coarse judgments from art-history students. On DOCENT, PoSh achieves a Spearman of 0.600 for overall quality, exceeding SPICE at 0.458, CAPTURE at 0.453, and GPT-4o Judge at 0.510, while also outperforming alternatives in mistake and omission localization (Ananthram et al., 21 Oct 2025).
IT proposes a complementary benchmark suite oriented toward generation quality, faithfulness, and linguistic richness. DID-Bench uses human-checked elaborate ground-truth descriptions and evaluates with BLEU, ROUGE-L, METEOR, SPICE, and WMD. D2I-Bench converts descriptions back into images with PixArt and compares them to original images using CLIP-score and DINO-score. LIN-Bench measures readability and descriptive richness through ARI, Flesch-Kincaid, and SMOG, while POPE probes hallucination rates through object existence queries. These benchmarks collectively treat description quality as simultaneously semantic, visual, and linguistic (Pi et al., 2024).
Accessibility-oriented evaluation introduces yet another criterion set. In a study focused on visually impaired students, expert interviews yielded requirements for short and long descriptions, including conciseness for alt-text, completeness and accuracy for long descriptions, semantic meaning beyond color or shape, navigability, gradation of detail, and screenreader friendliness. The resulting questionnaire contains 15 Likert-scale items scored in a two-stage process, first without the image and then with the image; the description score is
with a maximum score of 75 (Hoppe et al., 2021).
A persistent controversy in this area concerns whether standard caption metrics are still informative. The recent literature suggests that for long descriptions, they are often insufficient on their own. This suggests that DoI evaluation is moving toward decomposed assessment: localized span-level error attribution, omission-versus-mistake separation, faithfulness-by-regeneration, and downstream usability.
5. Downstream functions and multimodal applications
Detailed image descriptions are increasingly used not only as outputs, but also as supervision and control signals. IT-generated descriptions improve downstream description models: LLaVA-7B trained on IT-curated data produces richer descriptions, substantially increases the length and detail of its output, and exhibits less hallucination. On DID-Bench, IT descriptions outperform MLLM-generated descriptions by large margins; for example, on GT-{GPT4-V}, BLEU-1 increases from 45.26 to 57.38 and SPICE from 52.21 to 55.20 (Pi et al., 2024).
DCI demonstrates that dense, highly aligned text-image pairs expose weaknesses that are invisible on standard retrieval benchmarks. The dataset’s subcrop-caption matching task requires models to discriminate between very similar visual regions paired with detailed, localized summaries. Current models perform poorly on the hardest variants: CLIP reaches about 40% on SCM before finetuning, NegCLIP can reach up to 43% on All SCM, but on All Pick5-SCM top scores collapse to nearly chance, with CLIP at 11.3%. Finetuning CLIP on sDCI nevertheless yields significant improvements over the baseline despite the small training set (Urbanek et al., 2023).
IIW shows that hyper-detailed descriptions can function as unusually effective supervision. Human evaluations report major gains compared to recent datasets and GPT-4V across comprehensiveness, specificity, hallucinations, and related dimensions; the summary reports +66% relative to recent datasets and +48% relative to GPT4V. Models fine-tuned with IIW data improve side-by-side preference by +31% over models trained with prior work, even with only 9k samples. IIW-generated descriptions also produce the highest-fidelity text-to-image outputs and improve compositional reasoning by up to 6% on ARO, SVO-Probes, and Winoground (Garg et al., 2024).
DOCCI provides a similar training signal from a fully human-authored corpus. A PaLI 5B model finetuned on the DOCCI training set produces much longer descriptions than a COCO-finetuned counterpart and outperforms LLaVA-1.5 7B and InstructBLIP 7B on BLEU@4, ROUGE-L, METEOR, and CIDEr. The same dataset also exposes limits in text-to-image generation: DALL-E 3, Imagen, and SDXL all struggle with long descriptions and fine details, and CLIPScore is reported as unreliable for long, dense prompts because of text encoder truncation (Onoe et al., 2024).
DoI also serves as an explicit control modality for other vision tasks. MMC conditions image colorization on object-level and scene-level textual descriptions encoded by BERT, using Mask R-CNN segmentation, element-wise text-image feature fusion, and a fusion network to recombine colorized objects. The reported performance reaches LPIPS 0.120, PSNR 28.214, and SSIM 0.938, with ablation showing that removing textual conditioning degrades all three metrics (Ghosh et al., 2023). In text-to-image generation, AI Illustrator proposes a Prompt-based Cross-Modal Generation Framework that uses CLIP and StyleGAN, includes a projection module from text embeddings to image embeddings, and introduces a benchmark of 200 raw descriptions; the method is presented as handling complex descriptions without requiring external paired data for training (Ma et al., 2022).
6. Bias, accessibility, and descriptive infrastructure
DoI is often treated as a neutral rendering of visual facts, but empirical work shows that descriptions are rarely purely objective. In cross-linguistic datasets, annotators make inferences such as labeling women holding infants as “mother” or “grandmother,” and sometimes make stereotype-driven assumptions. Ethnicity or racial marking is disproportionately applied to non-white individuals in Dutch, English, and German descriptions, while “white” remains largely unmarked. Negation is rare but consistent across languages, with 27 English, 20 German, and 11 Dutch descriptions containing explicit negation in a corpus of 5,070 sentences per language (Miltenburg et al., 2017).
Accessibility research introduces a different normative standard. For visually impaired students, a useful description must be complete, correct, and easily understandable using a screenreader. Template-based automatic descriptions appear promising for structured illustrations such as bar charts and pie charts, but are less effective for node-link diagrams, where score differences relative to controls include , , and 0 points. The study therefore treats layered detail, HTML structuring, and semantic explanation of symbols and labels as central design criteria rather than optional stylistic refinements (Hoppe et al., 2021).
Descriptive practice also depends on metadata systems that outlive the generation event itself. DataCite DOI metadata records for datasets change frequently but usually incrementally: 89.05% of records registered in 2021 changed at least once within two years, or 80.78% when probable automated changes are excluded; the median is 2 versions and the mean is 2.23 versions per record, with a mean of 1.99 elements affected per change. Completeness increases only slightly after registration—11.7% of records gain new elements, and on average only 0.22 previously unused elements are added per changed record. For images and other digital objects, this means that initial metadata quality is critical, because later enrichment is uncommon and schema mapping can lose detail relative to richer local descriptions (Strecker, 2024).
Taken together, these findings define DoI as more than caption generation. It is a composite research area spanning annotation design, grounding, evaluation, accessibility, model supervision, and metadata maintenance. The current literature suggests that progress depends less on producing longer text per se than on producing descriptions that are discriminative, grounded, auditable, and appropriate to their downstream use.
