CLIPVehicle: Unified Vehicle Search Framework

• CLIPVehicle is a unified framework for vision-based vehicle search that optimizes both detection and re-identification using CLIP-derived semantic priors.
• The system uses a shared ResNet-50 backbone, region proposal networks, and multi-level identification learning to enhance detection accuracy and identity matching.
• It achieves improved mAP and Top-1 metrics on benchmarks like CityFlowVS and SynVS, demonstrating advantages over separate detection and re-ID pipelines.

CLIPVehicle is a unified framework for vision-based vehicle search that performs vehicle detection and vehicle re-identification directly on raw surveillance frames, so that a cropped query vehicle can be matched against detections in gallery video without a separate detect-then-store-then-Re-ID pipeline. It is formulated as an end-to-end system in which CLIP and CoOp-derived semantic priors are used to mitigate the tension between detection, which emphasizes vehicle commonness, and re-identification, which emphasizes individual uniqueness (Wang et al., 6 Aug 2025).

1. Task formulation and problem setting

CLIPVehicle defines vehicle search over annotated frames

$$\mathcal{D} = \{(\mathbf{x}_t, \mathbf{y}_t)\}_{t=1}^T,$$

where each frame $\mathbf{x}_t$ has annotations

$\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$

Here $b_t^j$ is the bounding box of the $j$-th vehicle in frame $\mathbf{x}_t$, $c_t^j \in \{1,\dots,C\}$ is its identity label, $N^t$ is the number of vehicles in that frame, and $C$ is the total number of identities in the training set. During evaluation, the input is a cropped query vehicle image $\mathbf{q}$ and a gallery of raw frames $\mathbf{x}_t$0. The system must detect vehicles in each gallery frame, extract identity embeddings for each detection, and rank all detections by similarity to the query embedding. A detection is counted as correct when its IoU with a ground-truth box of the same identity exceeds 0.5, and evaluation uses mAP and CMC Top-1. Training identities and test identities do not overlap (Wang et al., 6 Aug 2025).

The framework is motivated by a structural conflict in joint detection and Re-ID. Detection benefits from features that encode shared vehicle-level regularities, whereas Re-ID benefits from features that amplify subtle inter-instance differences such as color shade, local decorations, or viewpoint-dependent cues. Conventional vehicle search systems therefore pre-detect and store vehicle patches and then apply a separate Re-ID model, but this is resource-intensive, propagates detection errors into retrieval, and mismatches training on clean crops with deployment on noisy detections. CLIPVehicle is presented as the first unified vision-based vehicle search framework precisely to address this end-to-end formulation (Wang et al., 6 Aug 2025).

2. Backbone, detection pipeline, and identity branch

CLIPVehicle uses a shared ResNet-50 backbone up to conv4 to produce a feature map

$\mathbf{x}_t$1

A Region Proposal Network operates on $\mathbf{x}_t$2 to generate proposals $\mathbf{x}_t$3. For each proposal $\mathbf{x}_t$4, RoI Align extracts

$\mathbf{x}_t$5

and a detection-head conv block $\mathbf{x}_t$6, implemented as conv5 of ResNet-50, produces

$\mathbf{x}_t$7

A Faster R-CNN-style box predictor then outputs

$\mathbf{x}_t$8

This stage is summarized by

$\mathbf{x}_t$9

The Re-ID branch is sequentially conditioned on the predicted detections. Using $\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$0, RoI Align is applied again on the shared backbone feature map to produce $\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$1, which is then processed by a separate conv5 block $\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$2:

$\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$3

A regression head refines the box,

$\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$4

and a Norm-Aware Embedding head outputs both the identity embedding and a foreground/background score:

$\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$5

The embedding $\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$6 is trained with Online Instance Matching. The result is a sequential ReID-by-detection architecture with a shared backbone, a detection head for proposal refinement, and a distinct identity head for embedding learning (Wang et al., 6 Aug 2025).

3. Dual-granularity semantic-region alignment

A central component of CLIPVehicle is dual-granularity semantic-region alignment, which uses CLIP and CoOp to inject semantic supervision at both object and identity levels. At object granularity, the prompts are fixed as "A photo of a vehicle" and "Not a photo of a vehicle", and their CLIP text embeddings are compared with the region feature $\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$7. The foreground and background affinities are

$\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$8

where $\mathbf{y}_t = \{(b_t^j, c_t^j)\}_{j=1}^{N^t}.$9 is cosine similarity. With binary label $b_t^j$0, the object-level semantic-region alignment loss is

$b_t^j$1

This makes the detection representation semantically consistent with foreground-background structure.

At identity granularity, CLIPVehicle uses CoOp-style learnable prompts rather than raw ID strings. The prompt template is "A photo of a [X]_1 [X]_2 \cdots [X]_M vehicle", and an attribute-enriched variant extends it to "A photo of a [X]_1 [X]_2 \cdots [X]_M vehicle with [X] color and [X] type". After a pre-training stage, each identity class $b_t^j$2 is associated with a text embedding $b_t^j$3. For each ground-truth box feature $b_t^j$4, identity-granularity alignment is enforced by

$b_t^j$5

where $b_t^j$6 for the true identity and $b_t^j$7 otherwise. The total semantic alignment term is

$b_t^j$8

This gives CLIPVehicle a semantic grounding that operates simultaneously at the level of vehicle existence and individual vehicle identity (Wang et al., 6 Aug 2025).

4. Multi-level vehicle identification learning and optimization

To stabilize identity learning under noisy detections, CLIPVehicle introduces multi-level vehicle identification learning at image, box, and feature levels. At image level, the IDs present in frame $b_t^j$9 are converted to a binary vector $j$0, and a multi-label classifier $j$1 predicts $j$2 from the shared backbone feature map. The image-level loss is

$j$3

This supplies full-frame, multi-ID supervision.

At box level, ground-truth boxes are also passed through the ID head to obtain clean instance features, and a single-label classifier $j$4 predicts the identity:

$j$5

At feature level, a separate Re-ID teacher model is pre-trained on cropped vehicle patches and then frozen. For each ground-truth crop feature $j$6 from that teacher and corresponding CLIPVehicle feature $j$7, a consistency loss is applied:

$j$8

The multi-level identification term is therefore

$j$9

Training proceeds in two stages. First, the CoOp tokens are learned with frozen CLIP encoders on cropped vehicle patches, and the Re-ID teacher is trained separately. Second, the main CLIPVehicle model is optimized with the total objective

$\mathbf{x}_t$0

Implementation details include a ResNet-50 backbone, input resize to $\mathbf{x}_t$1, batch size 5, SGD optimization, RoI feature size $\mathbf{x}_t$2, $\mathbf{x}_t$3, $\mathbf{x}_t$4, and $\mathbf{x}_t$5 learnable CoOp tokens, on an NVIDIA RTX 3090 (Wang et al., 6 Aug 2025).

5. Benchmarks, datasets, and empirical profile

CLIPVehicle is evaluated on one real-world benchmark, CityFlowVS, and two synthetic benchmarks, SynVS-Day and SynVS-All. CityFlowVS is derived from CityFlowV2 by extracting all frames, discarding frames without vehicles, sampling one frame every five frames, inheriting bounding boxes and IDs, splitting scenarios S01-S04 for training and S05 for testing, and generating one cropped query per vehicle-camera pair in the test split. SynVS-Day and SynVS-All are constructed from Synthehicle, retaining only vehicles, removing vehicles that appear in only one scenario, training on Town01-Town03, and testing on Town04-Town05; SynVS-Day uses only daytime scenes, whereas SynVS-All uses day, dawn, rain, and night (Wang et al., 6 Aug 2025).

Dataset	Training split	Test split
CityFlowVS	329 IDs, 6,561 frames, 17,752 boxes	337 IDs, 11,550 frames, 32,877 boxes, 1,759 queries
SynVS-Day	475 IDs, 11,102 frames, 95,473 boxes	335 IDs, 6,810 frames, 49,925 boxes, 1,277 queries
SynVS-All	1,457 IDs, 44,273 frames, 388,087 boxes	843 IDs, 25,645 frames, 197,038 boxes, 4,038 queries

Against pure vehicle Re-ID baselines such as CLIP-ReID, MSINet, and MBR, and person-search baselines such as SeqNet, COAT, and OIMNet++, CLIPVehicle achieves the strongest reported end-to-end search results. On CityFlowVS it reaches 14.1 mAP and 83.8 Top-1, compared with 13.3/82.5 for COAT and 12.8/83.0 for OIMNet++. On SynVS-Day it reaches 32.4/84.1, compared with 31.6/83.0 for COAT and 30.0/83.1 for OIMNet++. On SynVS-All it reaches 24.6/83.0, compared with 23.7/81.3 for COAT and 22.1/80.8 for OIMNet++. Scene-wise results on SynVS-All show 32.6/85.0 in day, 36.2/84.2 in dawn, 32.1/85.5 in rain, and 21.6/77.3 in night, indicating persistent difficulty under nighttime conditions. The accompanying discussion attributes the gains to the semantic alignment and multi-level ID learning, while also noting that pure Re-ID models degrade when forced to operate on detector outputs rather than clean crops (Wang et al., 6 Aug 2025).

6. Position in vehicle-centric CLIP research, limitations, and outlook

CLIPVehicle sits within a broader vehicle-centric CLIP literature but occupies a distinct systems position. CLIP-SENet shows that CLIP-derived semantics can improve vehicle Re-ID without any text encoder by combining TinyCLIP image features with an Adaptive Fine-grained Enhancement Module, reaching 92.9% mAP and 98.7% Rank-1 on VeRi-776 (Lu et al., 24 Feb 2025). By contrast, prototypical contrastive learning for CLIP-based vehicle Re-ID argues that prompt learning is unnecessary for vehicle identities when the image encoder is directly fine-tuned with prototype losses, achieving 82.5 mAP and 97.1 Rank-1 on VeRi-776 in the supervised setting and 45.5 mAP and 90.7 Rank-1 in the unsupervised O2CAP setting (Li et al., 2023). At the deployment level, Clip4Retrofit demonstrates that CLIP-like semantics can be distilled into an EfficientNet-B3 plus MLP model with a 24.6 MB binary, 25.6 MB runtime memory, and about 35 ms per 300×300 frame on a 3 TOPS automotive NPU (Zhong et al., 23 May 2025). These results suggest that CLIPVehicle belongs to a larger trajectory in which CLIP-derived semantics are being specialized for retrieval, recognition, and edge deployment across transportation systems.

The limitations emphasized for CLIPVehicle are also system-level. The framework depends on the quality of the underlying vision-language semantics, can inherit synthetic-to-real domain gaps from its synthetic benchmarks, and still degrades markedly in night scenes. It also requires dense bounding-box and identity annotations, and its attribute schema for prompt construction remains relatively simple. The stated future directions are richer textual descriptions, multimodal inputs, domain adaptation, and further optimization toward large-scale real-time deployment (Wang et al., 6 Aug 2025). A plausible implication is that subsequent CLIPVehicle-like systems will combine end-to-end search with stronger open-vocabulary semantics, lighter deployment backbones, and cross-domain adaptation, rather than treating detection, retrieval, and semantic grounding as isolated subsystems.