T2I-VeRW: Part-level Fine-grained Perception for Text-to-Image Vehicle Retrieval

Abstract: Vehicle Re-identification (Re-ID) aims to retrieve the most similar image to a given query from images captured by non-overlapping cameras. Extending vehicle Re-ID from image-only queries to text-based queries enables retrieval in real-world scenarios where only a witness description of the target vehicle is available. In this paper, we propose PFCVR, a Part-level Fine-grained Cross-modal Vehicle Retrieval model for text-to-image vehicle re-identification. PFCVR constructs locally paired images and texts at the part level and introduces learnable part-query tokens that aggregate both part-specific and full-sentence context before aligning with visual part features. On top of this explicit local alignment, a bi-directional mask recovery module lets each modality reconstruct its masked content under the guidance of the other, implicitly bridging local correspondences into global feature alignment. Furthermore, we construct a new large-scale dataset called T2I-VeRW, which contains 14,668 images covering 1,796 vehicle identities with fine-grained part-level annotations. Experimental results on the T2I-VeRI dataset show that PFCVR achieves 29.2\% Rank-1 accuracy, improving over the best competing method by +3.7\% percentage points. On the newly proposed T2I-VeRW benchmark, PFCVR achieves 55.2\% Rank-1 accuracy, outperforming a comprehensive set of recent state-of-the-art methods. Source code will be released on https://github.com/Event-AHU/Neuromorphic_ReID

• The paper presents a novel PFCVR framework that uses explicit part-level alignment with Grounding DINO and learnable part-query tokens.
• It achieves significant improvements with a Rank-1 accuracy of 29.2% on T2I-VeRI and 55.2% on T2I-VeRW, surpassing previous models.
• The study offers a large-scale T2I-VeRW dataset featuring detailed part annotations, providing a new benchmark for cross-modal vehicle retrieval research.

T2I-VeRW: Part-level Fine-grained Perception for Text-to-Image Vehicle Retrieval

Introduction

Text-to-image vehicle retrieval, a critical cross-modal task in Intelligent Transportation Systems (ITS), has precise real-world motivation: surveillance environments often provide only a witness's textual vehicle description, necessitating retrieval engines capable of matching these descriptions to image databases. Existing cross-modal vehicle retrieval methods, typically leveraging holistic dual-encoder architectures or coarse spatial priors, are fundamentally limited at part-level granularity, with inadequate handling of local-global alignment and insufficient dataset scale for comprehensive part-based modeling. The paper "T2I-VeRW: Part-level Fine-grained Perception for Text-to-Image Vehicle Retrieval" (2605.06012) addresses these limitations by proposing the PFCVR framework, which explicitly models fine-grained vehicle components in a cross-modal alignment regime and introduces T2I-VeRW, a new large-scale dataset with rich part-level annotations.

Figure 1: Comparison of (a) and (b) existing text-based vehicle retrieval paradigms with (c) the proposed part-level fine-grained alignment framework.

PFCVR Framework: Architecture and Methodology

The PFCVR (Part-level Fine-grained Cross-modal Vehicle Retrieval) model innovates on two core limitations in prior work. First, it achieves explicit part-level alignment by localizing six predefined vehicle parts—windows, wheels, doors, mirrors, lights, and roof—using Grounding DINO. This enables the construction of locally paired image and textual part representations. Second, it introduces learnable, context-aggregating part-query tokens, ensuring local part-level semantics are embedded within global sentence context prior to alignment, thus resolving the inherent conflict between uncontextualized local tokens and the global alignment objective.

The high-level pipeline operates as follows:

• Vehicle part localization is performed in the image modality with Grounding DINO.
• Part-masked images and masked textual descriptions augment the inputs. Both global and per-part (pseudo) image and text representations are encoded via pre-trained CLIP (ViT-B/16, Text Transformer).
• The Part-level Local Fine-grained Alignment (PLFA) module aligns contextually enriched learnable part-query tokens with detected visual part regions using multi-head cross-attention.
• The Bidirectional Mask Recovery Implicit Alignment (BMRIA) module enforces that each modality (text/image) reconstructs its masked information, conditioned on features from the other, implementing both image-guided masked language modeling and text-guided masked image modeling.
• The global loss combines SDM, ID classification, part-level ITC, and bi-directional implicit reconstruction losses.

  Figure 2: Schematic of the PLFA and BMRIA modules, where PLFA provides explicit, contextually conditioned local alignment and BMRIA regularizes global structure via bidirectional mask recovery.

T2I-VeRW Dataset: Scale and Annotation Granularity

Prior datasets (notably T2I-VeRI) restrict the achievable generalization and fidelity of cross-modal vehicle retrieval models due to limited scale and sparse, weakly annotated examples. The introduced T2I-VeRW dataset comprises 14,668 images and 1,796 identities, marking a significant increase in inter-class and intra-class visual and textual diversity. Each sample contains granular, free-form descriptions (average 50.35 words, vs. 27.6 in T2I-VeRI) generated by large multimodal LMs and manually post-edited, explicitly detailing vehicle part attributes.

Segmentation masks for six parts are generated using the Segment Anything Model 3 (SAM3), stored as binary matrices, with systematic prompt engineering to maximize prompt-annotation coverage (e.g., covering ``windshield, side windows, rear window'' for the windows category). This segmentation approach offers pixel-level precision unmatched by prior bounding box or mask approximations.

Experimental Results

PFCVR demonstrates consistent SOTA performance, achieving 29.2% Rank-1 accuracy (mAP 25.3) on T2I-VeRI, exceeding the previous best by +3.7 points. On T2I-VeRW—the more challenging, larger, and more variegated benchmark—the framework secures 55.2% Rank-1, improving over strong baselines, including competitive 2025 era text-to-image person retrieval methods retrained on the vehicle domain. A detailed ablation verifies that both PLFA and BMRIA are necessary to realize these gains and their contribution is additive. Application of targeted data augmentation (gamma correction, Gaussian noise) further enhances generalization, particularly in the low-sample training regime.

The importance of explicitly contextualized part-level alignment and bidirectionally regularized global structure is highlighted by PLFA’s superiority over pure part MLM (masked language modeling on part tokens; PLFA achieves +2.9% higher Rank-1), as well as the sensitivity analysis regarding mask ratios and local alignment loss weights. These findings validate the assertion that prior person-centric cross-modal methods inadequately transfer due to their structural priors (pose, attribute ontology) insufficient for the rigid, symmetric structure of vehicles.

Model Limitations and Future Directions

Qualitative error analysis reveals confusion among vehicles sharing global attributes (e.g., body color, shape) but diverging in fine-grained details such as texture or emblem, indicating the current bounding-box-level part detection is insufficient for specific instance-level discrimination. Moreover, low-resolution input constraints in CLIP (384×384) impair the model’s ability to disambiguate informative, small-scale cues (stickers, badges).

Potential enhancements include integration of pixel-level or texture-centric keypoint descriptors, upscaling backbone resolution, or leveraging multimodal LLMs that can explicitly parse and relate texture descriptors in both modalities. Additionally, domain-specific regularization with license plate or location cues may mitigate ambiguity for highly similar instances.

Theoretical and Practical Implications

The main implication is that explicit modeling of parts and their context within cross-modal embeddings is essential for fine-grained vehicle retrieval. Part-query mechanisms, when executed with sentential conditioning, mediate between global and local objectives, mitigating the representation gap inherent in naïve region-word or keyword-only paradigms. Bidirectional implicit modeling—here as mask recovery—serves as a generalizable self-supervised signal enforcing robust cross-modal coupling. The efficacy across datasets of disparate scale further suggests that the approach generalizes beyond dataset artifacts, establishing a new methodological baseline for text-image vehicle re-identification.

Practically, these findings underpin the development of real-world retrieval tools deployable under incomplete evidence (i.e., only textual description), reducing error rates and operational load in law enforcement and transportation management. The T2I-VeRW dataset provides a robust and scalable public testbed for benchmarking future models.

Conclusion

The PFCVR framework, with its fusion of part-level, contextually aware alignment and bidirectional implicit regularization, directly addresses core bottlenecks in text-to-image vehicle retrieval. Empirical superiority on both legacy and newly constructed datasets coupled with robust ablations substantiates the premise that vehicle-specific component modeling is indispensable. While remaining challenges around texture-level disambiguation and resolution persist, this work materially expands the methodological toolkit available for cross-modal vehicle retrieval, indicating fertile ground for ongoing research at the intersection of region-based alignment, self-supervised learning, and advanced multimodal dataset curation.