Fine-class Described Retrieval Dataset (FDRD)

• FDRD is a suite of benchmarks that enable fine-grained retrieval through extensive subclass and attribute annotations in both image-text and 3D domains.
• It addresses limitations of prior datasets by incorporating detailed, semantically rich annotations and rigorous evaluation protocols to ensure precise attribute matching.
• Baseline results demonstrate significant improvements in retrieval accuracy, promoting advances in discriminative representation learning for vision-language and 3D shape tasks.

The Fine-class Described Retrieval Dataset (FDRD) refers to two independently developed, domain-specific benchmarks for fine-grained retrieval: (1) a large-scale image–text dataset designed for attribute- and subcategory-sensitive image–text retrieval tasks in computer vision and vision-language modeling (Wang et al., 6 Aug 2025), and (2) a 3D shape retrieval benchmark with complete subclass structure and part segmentation labels for evaluating shape-based retrieval under significant intra-class variation (Fu et al., 2020). Both are constructed to address the inadequacies of prior datasets that lack exhaustive category granularity or insufficient semantic annotation, providing rigorous protocols and metrics for evaluating methods on fine-class matching.

1. Motivation and Distinctive Attributes

The FDRD instances aim to advance retrieval methodology by enabling rigorous, fine-level evaluation. The image–text FDRD addresses the inability of datasets such as MSCOCO or Flickr30K to test attribute-level precision and subclass discrimination; captions in such datasets are predominantly coarse or lack linkage to discriminative features (e.g., pose, background, minute color or pattern) (Wang et al., 6 Aug 2025). In 3D shape retrieval, standard datasets like ModelNet and ShapeNet Core55 offer only class-level labels, omitting fine semantic distinctions and part-aware structure (Fu et al., 2020).

Both datasets are characterized by:

• Explicit fine-grained subclass labels spanning large category taxonomies.
• Rich, semantically granular annotations (detailed natural language attributes or part segmentations).
• Rigorous retrieval protocols that disallow ambiguous matching, requiring true fine-class discrimination.

2. Image–Text FDRD: Composition and Annotation Protocol

The image–text FDRD offers 1,523 fine categories from nine meta-domains (e.g., Caltech101, Flowers102, Food101, OxfordPets, StanfordCars, DTD, SUN397, UCF101, FGVC Aircraft), comprising 230,412 image–caption pairs (Wang et al., 6 Aug 2025). Each caption has an average length of 47.5 tokens and describes 5–7 visual attributes, systematically sampled from canonical attribute groups: color, pattern/marking, texture, shape/pose, and context/background.

The annotation workflow utilizes a three-stage pipeline with AI-generated drafts refined by feedback loops (InternLM-XComposer VLLM for initial captioning; Qwen2.5-VL for quality control), followed by human-in-the-loop correction for hallucinations and enforcement of style uniformity. This hybrid process yields high inter-annotator reliability (Cohen’s κ ≈ 0.82 for attribute correctness).

Train/val/test splits are generated using few-shot protocols: 16/4 images per category for train/val, with the remaining ≈30 per category in test (no overlap), enabling controlled comparison for models like CoOp/CoCoOp, and facilitating few-shot learning research.

3. 3D FDRD (RISA-Dataset): Structure and Semantic Partitioning

The 3D FDRD, released with RISA-Net, consists of 8,906 watertight mesh models distributed across six object categories (Car, Plane, Chair, Table, Guitar, Knife), each further divided into 175 manually curated sub-classes (Fu et al., 2020). Subclass assignment follows a strict protocol based on intra-class semantic features—product model, functionality, or design style—with secondary expert curation to discard ambiguous shapes and reinforce taxonomic coherence.

Semantic part segmentation is provided for each mesh, with all shapes pre-registered to a canonical reference (3,075 vertices/unit cube alignment). This guarantees topologically consistent meshes for part-based retrieval methods and enables edge/part feature extraction on a per-segment basis.

4. Data Format, Preprocessing, and Accessibility

Both FDRDs provide standardized data layouts and normalization schemes. The image–text FDRD is distributed with images named by unique IDs, JSON files for train/val/test splits, and explicit metadata including category hierarchies. Each JSON entry includes structured fields for image ID, file path, domain, category, free-text caption, and an attribute dictionary (e.g., color, texture) (Wang et al., 6 Aug 2025).

Image preprocessing follows the CLIP convention: 224 px short side, center crop to 224×224, pixel normalization (mean = [0.48145466, 0.4578275, 0.40821073], std = [0.26862954, 0.26130258, 0.27577711]). Captions are tokenized using CLIP's BPE and truncated to 77 tokens, while per-token attribute weights $w^{j}_{token}$ are computed for retrieval with dynamic reweighting modules.

The 3D dataset stores mesh models as watertight .obj/.off files, with per-part sub-meshes accompanying each shape. All models are SO(3)-perturbed in evaluation to assess rotation invariance, and retrieval splits follow a 4:1 train:test per category (≈7,125:1,781) (Fu et al., 2020).

Both datasets are publicly available with open licenses and established GitHub repositories.

5. Retrieval Tasks, Protocols, and Metrics

The image–text FDRD supports image-to-text (I2T) and text-to-image (T2I) retrieval. Each query (image or caption) is paired with one correct target. Main evaluation criteria are Recall@$K$:

$$\mathrm{Recall@}K = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{ \mathrm{rank}_i \le K\}$$

and Mean Reciprocal Rank (MRR):

$$\mathrm{MRR} = \frac{1}{N}\sum_{i=1}^N \frac{1}{\mathrm{rank}_i}$$

where $\mathrm{rank}_i$ is the position of the correct item for query $i$ (Wang et al., 6 Aug 2025).

For the 3D FDRD, each test-shape is used as a query against a gallery of all shapes (category-matched, train+test), and a retrieval is “correct” only if the returned shape shares the query’s sub-class label. Metrics include Nearest Neighbor (NN), First Tier (FT), Second Tier (ST), Normalized Discounted Cumulative Gain (NDCG), and mean Average Precision (mAP), with micro- and macro-averaging across sub-classes:

• NN: proportion of queries for which the closest match is of the same sub-class.
• FT/ST: proportion of relevant items within top $|G_q|$ and $2|G_q|$ retrieved results.
• mAP: standard average precision per query, averaged over all queries.
• NDCG: rewards early correct retrievals in the ranked list (Fu et al., 2020).

6. Baseline Results and Comparative Analysis

For the image–text FDRD, benchmark results under 16-shot fine-tuning highlight the challenge and discriminative power of the dataset. In averaged Recall@1:

Method	I2T R@1	T2I R@1
CLIP (zero-shot)	27.04%	20.62%
CoOp	33.75%	18.26%
MaPLe	35.42%	19.12%
FILIP	29.98%	21.59%
FineCLIP	33.68%	24.63%
DCAR (Ours)	37.65%	27.25%

A full domain-wise breakdown shows that DCAR, a dual-prompt learning framework, outperforms all baselines by 3–11 points on Recall@1 (Wang et al., 6 Aug 2025).

For the 3D FDRD, RISA-Net achieves state-of-the-art micro-averaged retrieval (NN = 0.7378, FT = 0.5670, ST = 0.6892, NDCG = 0.7878, mAP = 0.6125) significantly surpassing methods such as MeshCNN and MVCNN (Fu et al., 2020). The protocol’s rotation perturbation and fine-class labeling drive these methods to encode both geometric and semantic detail.

7. Guidelines for Use and Extensions

For the image–text FDRD, best practices include adhering to the provided few-shot splits, using conservative image augmentations, deploying CLIP-ViT-B/32 for initialization with prompt-based adaptation, and evaluating on test splits without re-sampling. Results should be reported as Recall@1/5/10 and MRR for both I2T and T2I queries.

Future extensions for image–text FDRD include incorporating bounding boxes or token-level linguistic tags to increase attribute alignment granularity.

The 3D FDRD recommends strict adherence to the rotation protocol and labeled splits. Semantic segmentation facilitates direct use in part-based deep learning pipelines, eliminating the need for external part-labeling tools.

8. Significance and Impact

The FDRD datasets, in both image–text and 3D forms, provide rigorous, high-precision testbeds for advancing fine-grained retrieval. By enabling controlled, subcategory- and attribute-level evaluation, they expose deficiencies in existing methodologies, stimulate research in discriminative representation learning, and serve as benchmarks for both few-shot and rotationally invariant retrieval scenarios. They stand as critical resources for research in vision-language modeling (Wang et al., 6 Aug 2025) and 3D geometry-based retrieval (Fu et al., 2020).