Visual Semantic Descriptions (VSD)

• Visual Semantic Descriptions (VSD) are approaches that convert visual content into detailed natural language, exposing objects, relations, and actions.
• They cover tasks from object-pair-conditioned spatial descriptions and semantic anchors for retrieval to brain-decoded captions and interpretability of latent features.
• VSD methods employ advanced models like Transformer encoder-decoders, 3D scene graphs, and neural decoding pipelines, achieving significant gains in generation and retrieval metrics.

Visual Semantic Descriptions (VSD) denotes a family of representations and tasks in which visual content is rendered into linguistically explicit semantics. In the literature, the acronym is used in multiple senses: object-pair-conditioned spatial descriptions of images, MLLM-generated semantic anchors for image-text matching, entity- or filter-level visual explanations, and captions decoded from brain activity evoked by visual stimuli. Across these formulations, the shared objective is to move beyond labels or raw embeddings toward natural-language descriptions that expose objects, relations, actions, scene structure, or latent semantics in a form usable for generation, retrieval, interpretation, and analysis (Zhao et al., 2022, Chen et al., 11 Jul 2025, Matsuo et al., 2018).

1. Conceptual scope and historical lineages

A useful way to situate VSD is to distinguish between descriptive granularity and computational role. Some work uses descriptions as outputs of an image-to-text system, some uses them as intermediate semantic anchors for cross-modal alignment, and some uses them to explain internal representations or neural signals. This suggests that VSD is better understood as a methodological family than as a single standardized task.

Formulation	Representative works	Descriptive unit
Region/image description	(Karpathy et al., 2014)	Image regions and contiguous phrases
Visual Spatial Description	(Zhao et al., 2022, Zhao et al., 2023, Jin et al., 2024)	Object pair and spatial sentence
Semantic anchors for matching/retrieval	(Chen et al., 11 Jul 2025, Meng et al., 24 May 2025, Han et al., 2023)	Image-, entity-, or class-level descriptors
Interpretability and domain abstraction	(Yang et al., 2022, Tas et al., 2023)	Filters or architectural structures
Brain-decoded semantics	(Matsuo et al., 2018, Feng et al., 15 Mar 2025)	fMRI-to-caption mappings

Foundational work on visual-semantic alignment already treated description as a region-sensitive correspondence problem rather than a whole-image captioning problem. “Deep Visual-Semantic Alignments for Generating Image Descriptions” learns alignments between image regions and sentence fragments from weakly supervised image-caption pairs, uses the top 19 detected locations plus the whole image, embeds words with a BRNN, smooths alignments with an MRF, and trains a Multimodal RNN for full-image and region-level description generation (Karpathy et al., 2014). Earlier survey work on the complementary language-to-visualization direction described systems that generate “visual interpretations of the semantic content of a given natural language” as a static scene or a dynamic animation, organized into text-to-picture, text-to-scene, and text-to-animation categories (Hassani et al., 2016). Taken together, these lines establish the broader grounding problem within which later VSD formulations operate.

2. Spatially controlled image-to-text generation

One major usage of VSD is explicitly defined as Visual Spatial Description. In this formulation, “given an image $I$ and an object pair $\left<O_1, O_2\right>$ inside $I$, the VSD aims to output a word sequence $S=\{w_1,...,w_n\}$ to describe the spatial semantics between $O_1$ and $O_2$.” The task is object-pair conditioned, differs from generic image captioning, and is contrasted with Visual Spatial Relationship Classification (VSRC), which predicts one relation from a closed set such as on, in, next to, under, above, behind, in front of, to the left of, and to the right of (Zhao et al., 2022).

The benchmark introduced for this task contains 29K images and 143K descriptions. Its split statistics are 20,490 training images with 116,791 sentences, 2,927 development images with 16,823 sentences, and 5,855 test images with 10,038 sentences; average sentence lengths are 7.35, 7.33, and 8.04 tokens, respectively. The baseline models use Transformer encoder-decoder architectures with VL-BART and VL-T5 backbones, Faster R-CNN visual features, cross-entropy training, and beam search decoding. The work also studies pipeline and joint end-to-end integration of VSRC into VSD, concluding that “the joint end-to-end architecture is the better choice for their integration.” Among the main models, VL-T5 + VSRC-end2end achieves BLEU-4 54.31 and SPICE 68.74, while oracle spatial labels yield much larger gains, with VL-T5 + VSRC-golden reaching BLEU-4 72.12 and SPICE 77.29 (Zhao et al., 2022).

A later extension argues that 2D geometry is insufficient for VSD because first-person perspective can bias spatial interpretation. “Generating Visual Spatial Description via Holistic 3D Scene Understanding” introduces 3Dvsd, an encoder-decoder framework that uses an external 3D scene extractor, constructs a target object-centered 3D spatial scene graph Go3D-S$^2$G, encodes it with OcGCN, and samples diverse local structures with the scene subgraph selecting mechanism S$^3$ (Zhao et al., 2023). The model maps relative 3D positions into 26 universal direction terms, fuses graph features with image and text prompt representations, and reports strong gains on VSD-v1 and VSD-v2. On VSD-v1, it reaches BLEU-4 54.85, METEOR 43.25, ROUGE 79.38, CIDEr 483.05, and SPICE 68.76; on VSD-v2, BLEU-4 26.40, METEOR 26.87, ROUGE 55.76, CIDEr 272.93, and SPICE 46.97. On a harder VSD-v2 subset with overlapped and irregularly posed objects, it achieves BLEU-4 39.29, METEOR 34.27, ROUGE 67.88, CIDEr 328.56, and SPICE 55.42 (Zhao et al., 2023).

An abstract-level 2024 extension, LLaVA-VSD, frames VSD as classification, description, and open-ended description of visual spatial relationships. It constructs a VSD instruction-following dataset from figure-caption pairs, uses LoRA to fine-tune a 13 billion parameter Large Language and Vision Assistant that supports high-resolution images, and applies Qwen-2 to refine generated sentences for diversity and accuracy (Jin et al., 2024).

3. Semantic anchors for image-text matching and retrieval

Another influential use of VSD treats descriptions not primarily as end outputs, but as alignment objects that make image representations more linguistically explicit. “Visual Semantic Description Generation with MLLMs for Image-Text Matching” proposes MLLMs as visual semantic parsers that generate VSD as a semantic anchor bridging continuous, high-dimensional image features and discrete, structured text (Chen et al., 11 Jul 2025). The prompt requests a “comprehensive visual semantic description (15-30 words)” capturing the main scene, key objects, primary action, and significant visual details. Training uses MiniCPMV2.6-8B for the main variant and Florence-2-large-ft-0.77B for a lightweight variant; VSD is encoded with BGE-large-v1.5.

The architecture combines instance-level alignment and prototype-level alignment. At instance level, image features $\mathbf{v}$ are fused with pooled VSD embeddings $\tilde{\mathbf{d}}_{\text{vsd}}$ through a sigmoid gate,

$\left<O_1, O_2\right>$0

and optimized with a hard-negative triplet loss. At prototype level, K-means clusters VSD embeddings into semantic prototypes, images and texts are projected into prototype space, and Sinkhorn soft assignments define the alignment targets. The full objective is

$\left<O_1, O_2\right>$1

The method is designed as a plug-and-play enhancement for GPO, HREM, and CLIP ViT-B/32, and reports large retrieval gains on Flickr30K and MSCOCO. For example, GPO + VSD$\left<O_1, O_2\right>$2 on Flickr30K improves I2T R@1 from 81.7 to 86.1, T2I R@1 from 61.4 to 71.9, and rSum from 513.5 to 540.8. It also shows zero-shot gains on N24News and RSITMD (Chen et al., 11 Jul 2025).

A closely related retrieval formulation is Entity Visual Descriptions (EVDs). “EvdCLIP” defines an EVD as “a list of visual descriptions associated with an entity,” emphasizing color, shape, parts, quantity, and other appearance-specific features; the EVD knowledge base covers around 10,000 entities (Meng et al., 24 May 2025). EVDs are generated offline from visual entities extracted from datasets such as Flickr30k and MSCOCO, then injected into queries. Because naive concatenation introduces noise, the method trains an EVD-aware Rewriter (EaRW) initialized from T5-large, first with supervised fine-tuning and then with Preference Rank Optimization (PRO). During CLIP fine-tuning, query rewriting is applied with probability $\left<O_1, O_2\right>$3, and final inference averages the original and EVD-enhanced scores. On Flickr30K, CLIP improves from 64.8 to 66.9 on I2T R@1 and from 49.2 to 52.0 on T2I R@1; on MSCOCO, I2T R@1 improves from 43.7 to 46.8 and T2I R@1 from 32.7 to 35.2 (Meng et al., 24 May 2025).

A third retrieval-oriented line treats class descriptors as optimized visual semantic descriptions. “LLMs as Visual Explainers” casts descriptor design as a training-free optimization problem over CLIP-style prompt ensembling, with an LLM agent that iteratively refines descriptors using an improved confusion matrix, memory banks, mutation, crossover, and selection (Han et al., 2023). Evaluated on ImageNet, EuroSAT, UCF101, SUN, Caltech, DTD, CIFAR-10, Flowers102, and CUB, it reports an average top-1 accuracy of 68.09%, compared with 61.69% for CLIP, 64.16% for DCLIP, 64.00% for WaffleCLIP, and 64.62% for CuPL (Han et al., 2023).

4. Interpretability and domain-specific semantic abstractions

In interpretability research, VSD often means textual explanations of latent visual representations. LaViSE maps CNN filter responses into a semantic word space and generates filter-level descriptions without semantic supervision in the target model’s training data (Yang et al., 2022). The explainer learns a mapping

$\left<O_1, O_2\right>$4

using a hinge rank loss over pretrained semantic embeddings, then uses filter attention based on cosine similarity to aggregate distributed evidence across filters. At inference, it selects the top-$\left<O_1, O_2\right>$5 images activating a filter, retrieves the top-$\left<O_1, O_2\right>$6 words per image, and aggregates them into textual explanations. Evaluation spans MS COCO, Visual Genome, Broden, and PoP, combining human judgment with an IoU-based recall protocol in which a concept counts as ground truth when IoU > 0.04. The method also supports unsupervised dataset bias analysis through filter explanations and group-wise concept ratios (Yang et al., 2022).

Domain-specific VSD can also mean semantic abstraction targeted to a structured application domain. “Unveiling Spaces” addresses populated interior scenes and seeks “architecturally meaningful semantic descriptions” rather than generic scene labels (Tas et al., 2023). It generates 1000 virtual classrooms, randomizes room geometry, door/window positions, camera pose, and clutter, and trains a Pix2Pix conditional GAN to translate rendered images into architectural abstractions in which walls, ceilings, and floors are encoded as red, green, and blue, with a separate depth model. Three models are trained: $\left<O_1, O_2\right>$7 for architectural class labels from synthetic imagery, $\left<O_1, O_2\right>$8 for style-augmented synthetic imagery, and $\left<O_1, O_2\right>$9 for depth. On synthetic test data, $I$0 reports scores between 0.792 and 0.956, including 0.904 at 20,000 steps; $I$1 reaches mean 0.940, minimum 0.879, maximum 0.959; $I$2 has range 0.30 to 0.97 and average 0.79. Real photographs remain challenging, but style-adapting real photos toward the synthetic distribution improves performance more than training-time augmentation alone (Tas et al., 2023).

5. Brain-decoded visual semantics

A distinct neuroscience usage of VSD concerns generation of sentence-level descriptions from brain activity evoked by visual stimuli. “Describing Semantic Representations of Brain Activity Evoked by Visual Stimuli” uses a two-stage transfer-learning pipeline in which fMRI voxel vectors $I$3 are mapped to predicted VGG image features $I$4 and then decoded by a pre-trained image-captioning model (Matsuo et al., 2018). The caption model is trained on Microsoft COCO with 414,113 image-caption pairs and a vocabulary of 3,469 frequent words. The brain-to-feature model predicts a 4096-dimensional VGG feature from up to 65,665 voxels in the main setting, with 4,500 paired samples and 7,540 unlabeled samples for stacked-autoencoder pretraining of the 5-layer DNN. Three regressors are compared—ridge regression, a 3-layer neural network, and a 5-layer DNN—and the three-layer neural network performs best, while the 5-layer DNN overfits. The image-caption model alone reaches perplexity around 2.5. Using whole cortex often works better than reduced visual subsets, leading to the conclusion that semantic information for sentence generation is widespread across the cortex (Matsuo et al., 2018).

A more direct brain-to-text paradigm is introduced in “From Eye to Mind: brain2text Decoding Reveals the Neural Mechanisms of Visual Semantic Processing” (Feng et al., 15 Mar 2025). This model uses the Natural Scenes Dataset, restricted to 4 subjects (subj01, subj02, subj05, subj07), with 10,000 COCO images per subject, 9,000 for training and 1,000 shared images for testing. The pipeline consists of subject-specific brain tokenization, a shared transformer-based brain encoder, prediction of a 1536-dimensional latent semantic vector, and text inversion with Vec2Text. Training uses MSE, 200 epochs, AdamW, a one-cycle schedule with maximum learning rate 1e-3, and approximately 124M parameters for single-subject decoding or 156M parameters for four-subject training. Evaluation includes BLEU-1/2/3/4, METEOR, ROUGE, CIDEr, SPICE, CLIP-S, and RefCLIP-S. Neuroanatomical analyses show that higher-level ROIs outperform early visual cortex, and SHAP highlights MT+ complex, ventral stream visual cortex, and inferior parietal cortex as the strongest contributors to semantic decoding (Feng et al., 15 Mar 2025).

6. Evaluation protocols, misconceptions, and open issues

The evaluation of VSD depends strongly on which formulation is under study. Spatial and brain-decoding work relies heavily on text-generation metrics such as BLEU-4, METEOR, ROUGE, CIDEr, and SPICE; retrieval-oriented VSD emphasizes R@1, R@5, R@10, and rSum; interpretability-oriented systems use precision, prefer, IoU, and recall; brain-to-text work increasingly supplements lexical overlap with semantic metrics such as CLIP-S and RefCLIP-S (Zhao et al., 2022, Chen et al., 11 Jul 2025, Yang et al., 2022, Feng et al., 15 Mar 2025). The persistence of human evaluation in several lines suggests that automatic metrics alone do not fully capture whether a description is spatially accurate, semantically appropriate, useful, or faithful.

A common misconception is that VSD is simply another name for ordinary captioning. The spatial-description literature explicitly rejects that equivalence: standard image captioning is a summarized description of the whole image, whereas VSD is object-pair conditioned and focused on spatial semantics; VSRC is a shallow classification task, while VSD is a deep generation task (Zhao et al., 2022). Retrieval-oriented VSD likewise differs from standard dual-encoder alignment by inserting a semantic bridge between visual and textual representations rather than relying on surface-level similarity alone (Chen et al., 11 Jul 2025). Interpretability-oriented VSD differs again, because it describes filters or latent concepts rather than scenes themselves (Yang et al., 2022).

Limitations are correspondingly heterogeneous. Spatial VSD datasets currently cover only nine spatial relations and often provide only one sentence per instance (Zhao et al., 2022). Holistic 3D spatial description depends on an external 3D scene extractor pretrained on SUN RGB-D, producing domain shift for outdoor images (Zhao et al., 2023). Architectural VSD generalizes imperfectly from synthetic classrooms to real photographs (Tas et al., 2023). LaViSE requires a reference dataset with concept labels and masks, and lower layers remain difficult to explain (Yang et al., 2022). Brain-decoding work still contends with small paired datasets, proxy references, limited subjects, and overfitting in deeper regressors (Matsuo et al., 2018, Feng et al., 15 Mar 2025). MLLM-based matching approaches incur preprocessing cost and depend on the quality of generated semantic parses (Chen et al., 11 Jul 2025).

These divergences suggest that VSD is best viewed as a unifying research motif: the use of natural-language semantics to expose structure that is visually present but computationally implicit. Whether the target is a spatial relation, an entity-rich retrieval query, a CNN filter, an architectural envelope, or an fMRI response, the core wager is the same: language can act as a higher-level interface to visual meaning.