GEPT Picture-Description Dataset

• GEPT Picture-Description Dataset is a specialized linguistic resource that captures semantic, syntactic, and spatio-temporal features from human-generated picture descriptions.
• It employs rigorous crowdsourcing methods and transformer-based pipelines to ensure high-quality, diverse annotations for robust computational analysis.
• The dataset underpins clinical assessments of cognitive impairment by extracting precise spatio-semantic metrics and narrative structure from detailed image descriptions.

The GEPT Picture-Description Dataset is a specialized linguistic resource developed to support the automated and clinical assessment of cognitive-linguistic function through the analysis of human-generated picture descriptions. Closely related to protocols and datasets such as ABSTRACT-50S, PASCAL-50S, and the "Cookie Theft" description task, the GEPT dataset is designed to comprehensively capture the semantic, syntactic, and spatio-temporal properties of spoken picture descriptions, facilitating advanced analytic pipelines in both research and clinical contexts.

1. Data Collection Methodology

The structural and procedural approaches adopted in the GEPT Picture-Description Dataset align with best practices established in large-scale image description collection, as exemplified by the methodology presented in "Collecting Image Description Datasets using Crowdsourcing" (Vedantam et al., 2014). Data acquisition typically uses crowdsourcing platforms such as Amazon Mechanical Turk, imposing rigorous participant selection criteria to ensure annotation quality: contributors are restricted to specified geographic regions (e.g., United States) and must pass stringent thresholds for prior task approval rates (≥95%) and experience (≥500 approved HITs).

Annotation interfaces require each subject to "transcribe" the contents of a presented image, a specific instruction that emphasizes objective report over creative or inferential description. The interface integrates explicit rejection criteria for responses that deviate from grammaticality, objectivity, or content relevance, maintaining dataset integrity and reproducibility. Each image receives multiple, independent annotations from distinct individuals, optimizing linguistic and perspectival diversity.

2. Dataset Properties and Statistical Analysis

A distinctive feature of the GEPT-style approach is the high density of human annotations per image. In line with ABSTRACT-50S and PASCAL-50S, this entails collecting 50 human-written sentences for each picture, enabling robust analysis of linguistic variability and consensus.

Key statistical properties echo those meticulously detailed in reference datasets:

• For real photographic images (PASCAL-50S analogue), average sentence lengths approximate 8.8 words; for abstract, clipart-like images (ABSTRACT-50S analogue), this increases to roughly 10.59 words, a difference attributed to the "tighter semantic sampling" required for disambiguating such scenes.
• The mean sentence length $L_{avg}$ is quantified as $L_{avg} = (\sum_{i=1}^{N} w_i)/N$, where $w_i$ is the word count per description and $N=50$.

Linguistic diversity across annotators is a central property, directly supporting downstream machine learning and psycholinguistic analyses. The diversity, coupled with grounded content-focused instructions, aims to maximize the utility of the dataset for computational models requiring exposure to a plurality of valid human interpretations.

3. Automated Spatio-Semantic Feature Extraction

Recent methodologies have demonstrated the feasibility and efficacy of automatic extraction and ordering of content information units (CIUs) from picture descriptions using transformer-based LLMs. A pipeline developed in "Advancing Automated Spatio-Semantic Analysis in Picture Description Using LLMs" (Ng et al., 30 Sep 2025) exemplifies this paradigm, leveraging a BERT (bert-base-uncased) backbone to encode input sentences and output logit scores over predefined CIU classes.

The primary objective is multi-label classification of CIUs within each utterance:

• Binary cross-entropy loss is used to optimize detection:

$$L_{BCE} = -\frac{1}{K} \sum_{k=1}^K\left( y_k \log \sigma(s_k) + (1-y_k)\log(1-\sigma(s_k)) \right)$$

where $K$ is the number of CIU classes (e.g., 23), $y_k$ are binary ground truth labels.

To capture the sequential order of CIUs—critical for modeling narrative structure—an auxiliary pairwise ranking loss is used:

$$L_{rank} = \frac{1}{N} \sum_{i < j} \max(0, s_j - s_i + m)$$

where $s_i, s_j$ are logit scores, $m$ is the margin (set to 1), and $N$ is the number of CIU pairs. The total loss is a weighted combination: $L = (1-\lambda) L_{BCE} + \lambda L_{rank}$ (with $\lambda = 0.1$).

4. Evaluation Metrics and Comparative Performance

Model performance is assessed using 5-fold cross-validation, with group-based splits to preclude speaker leakage. The pipeline attains median precision of 93% and median recall of 96% for CIU detection. Sequence error rates—quantified via Levenshtein distance—stand at 24%, reflecting moderate preservation of the ground-truth narrative order.

Benchmarking against dictionary-based extraction methods, the BERT-based approach achieves higher Pearson correlations with ground truth spatio-semantic features. For example, standard deviation of the X-coordinate improves from 0.61 to 0.90, and corresponding gains are observed for metrics such as self cycles and cross-quadrant ratios. ANCOVA validates the clinical utility of these features, as F-values derived from BERT-extracted CIUs closely match those from manual annotation, unlike dictionary-based approaches which manifest increased variability and repetitive over-tagging.

5. Applications in Cognitive Impairment Assessment

The combination of dense human annotations and automated spatio-semantic analysis facilitates advanced applications in clinical speech analytics. The pipeline enables extraction of features such as total visual path distance, unique node count, and cycle repetition—each of which indexes deficits in visuospatial processing or narrative organization, known markers of cognitive-linguistic impairment (e.g., in MCI or Alzheimer's disease).

For the GEPT Picture-Description Dataset, this enables quantitative, scalable cognitive assessment without the labor intensiveness of manual annotation or the limitations of dictionary-based methods, accelerating research and clinical diagnostics. The method is particularly adaptable for diverse populations, as it does not rely on rigid vocabulary lists and can thus accommodate significant linguistic heterogeneity.

6. Open-Source Availability and Research Impact

As part of the reproducibility and adoption strategy, the described pipeline, including BERT model weights and training code, is open-sourced (Ng et al., 30 Sep 2025). This open access enables researchers to replicate, validate, and extend spatio-semantic analyses across datasets including GEPT, PROMPT, and others.

The precedent set by the design choices in earlier image description datasets (e.g., instructions, interface controls, diversity maximization) and the demonstrated efficacy of language-model-based analytic pipelines position the GEPT Picture-Description Dataset as a central resource for both technical development and translational clinical research.

7. Context Within the Landscape of Picture-Description Resources

While not the first to assemble large-scale, richly annotated image description corpora, the GEPT approach extends the rigor and annotation density of pioneering efforts such as the UIUC Pascal Sentence Dataset, ABSTRACT-50S, and PASCAL-50S (Vedantam et al., 2014), and leverages recent advances in automated language understanding (Ng et al., 30 Sep 2025). Its focus on capturing not only descriptive breadth but also narrative structure and spatio-semantic content renders it particularly suitable for the advancement of cognitive-linguistic analysis and automated clinical screening methodologies.