Revisiting Vehicle Color Recognition in Long-Tailed Surveillance Scenarios

Abstract: Vehicle color recognition is an important cue for vehicle identification in surveillance systems, especially when license plates are illegible due to low resolution, occlusion, motion blur, or poor illumination. However, real-world vehicle color distributions are highly imbalanced, making overall accuracy insufficient to assess performance on rare but operationally relevant colors. This paper presents a comprehensive study of vehicle color recognition under severe class imbalance using UFPR-VeSV, a challenging real-world surveillance dataset. We investigate synthetic minority-class augmentation through two off-the-shelf generative strategies: text-conditioned image generation with RunDiffusion/JuggernautXL and image-conditioned color editing with Gemini 2.0 Flash. The curated synthetic data are combined with modern visual representations, loss reweighting, learning-rate scheduling, color-safe augmentation, foreground-aware preprocessing, and ensemble fusion. The bestperforming approach achieves 94.6% micro accuracy and 79.7% macro accuracy, improving macro accuracy by 8.2 percentage points over recent literature. A manual error analysis further shows that many remaining failures are visually ambiguous even for human annotators, highlighting the practical limits of color-based vehicle identification in unconstrained surveillance imagery. The generated images and source code are publicly available at https://github.com/viniciusorru/vcr-synthetic

• The paper introduces synthetic augmentation methods using Gemini and text-conditioned generation to enhance minority-class vehicle color recognition on imbalanced datasets.
• It employs advanced vision backbones and tailored training strategies, achieving an 8.2pp macro-accuracy improvement to a best score of 79.7% with ensemble models.
• An in-depth error analysis reveals that inherent visual ambiguities in surveillance imagery set a practical performance ceiling despite algorithmic advances.

Vehicle Color Recognition in Long-Tailed Surveillance Scenarios

Introduction

Vehicle color recognition (VCR) remains a critical auxiliary to ALPR in intelligent transportation systems (ITS), augmenting vehicle identification under suboptimal surveillance conditions where license plates are occluded or degraded. Despite substantial progress with deep learning on controlled benchmarks, performance substantially degrades in real-world settings due to intra-class variation, adverse imaging conditions, and, most crucially, severe long-tailed class distributions endemic to vehicle color categories. This paper systematically addresses VCR under such operational constraints, leveraging UFPR-VeSV—a large, highly unbalanced, and realistic surveillance dataset—to probe the upper bounds of VCR capability under practical constraints (2606.13625).

Characterization of the UFPR-VeSV Dataset and its Imbalance

The UFPR-VeSV dataset anchors the empirical analysis, featuring 24,945 images of 16,297 vehicles annotated for 13 color classes. The class distribution is acutely skewed: "white" dominates, while colors such as "brown" and "orange" are extreme minorities.

Figure 1: Distribution of vehicle colors in UFPR-VeSV reveals a strongly long-tailed nature, with white, black, silver, and gray overwhelmingly outnumbering other colors.

This imbalance trivializes micro-accuracy as an evaluation criterion and emphasizes macro-level metrics—macro-accuracy and macro F1-score—for meaningful assessment of rare color class predictions.

Synthetic Data for Minority-Class Augmentation

Synthetic augmentation strategies are central to the study. Two off-the-shelf generative approaches are employed:

1. Text-conditioned generation: RunDiffusion/Juggernaut-XL, a photorealistic diffusion model derived from Stable Diffusion XL, generates vehicles with arbitrary color, make, and scene attributes, with prompts diversified across underrepresented classes.

Figure 2: Examples of synthetic vehicles by RunDiffusion/Juggernaut-XL, covering minority and majority color classes across varied vehicle types, perspectives, and scenes.

2. Image-conditioned color editing: Gemini 2.0 Flash modifies existing real images to match target color prompts while preserving vehicle geometry, background, and imaging artifacts. The protocol carefully avoids data leakage by restricting edited images to train folds only.

Figure 3: Gemini-based image edits, showing original vehicle images (UFPR-VeSV) and their synthetic counterparts after color-targeted modification.

Manual, two-annotator curation is mandatory for realism, correct color, and domain fit. Inter-rater results indicate moderate agreement, especially given ambiguous boundary cases (e.g., gray vs. silver, beige vs. brown).

Recognition Backbone, Optimization, and Pipeline

Four supervised vision backbones—EfficientNet-V2, ResNet-50, Swin Transformer Tiny, and ViT-B/16—are benchmarked alongside DINOv3 foundation models, which offer frozen, transferable representations for robust feature extraction. For classifier heads, two-layer MLPs are used over [CLS] tokens. Optimization leverages Adam, early stopping, and three LR schedulers (Cosine Decay (CD), Linear Warmup + CD, ReduceLROnPlateau). Class imbalance is explicitly targeted via weighted cross-entropy (WCE), which is found to be unstable without synthetic balancing data in long-tailed setups.

Data augmentation pipelines—color-safe by design—avoid transformations that corrupt semantic color information. Foreground-aware preprocessing via SAM~2 segments the vehicle, blurring background to reduce scene bias.

Figure 4: Foreground-aware (SAM~2-based) preprocessing, highlighting vehicles and blurring backgrounds to focus the model on color-relevant evidence.

Model fusion is performed via hard voting ensembles among top-performing pipelines.

Numerical Results and Ablation Study

The strongest previous macro-accuracy on UFPR-VeSV was 71.5%. DINOv3-Large reaches 72.5% without synthetic data, with all baseline methods clustered near this limit. Incorporating Gemini-augmented data and WCE enables stable improvements: macro-accuracy rises to 75.7%. Layered on this, scheduler tuning (LWCD), color-safe augmentation, and foreground pre-processing each add 1–2% gains, yielding best single-model macro-accuracy of 78.1%. Majority voting ensembles reach 79.7% macro-accuracy and 94.6% micro-accuracy—an 8.2pp macro-accuracy improvement over prior art.

Notably, models augmented with Gemini, although comprising fewer images than RunDiffusion, outperform due to better domain alignment and preservation of surveillance scene priors. This emphasizes that synthetic data diversity and contextual fidelity trump sheer volume when mitigating minority-class scarcity.

Error Analysis and Operational Boundaries

A fine-grained manual analysis of erroneously classified test images uncovers that over half the errors are visually ambiguous, often irreducible even for expert annotators. These ambiguous misclassifications are concentrated among similar neutral tones and images with limited color evidence due to poor illumination, occlusions, and non-visible vehicle regions.

Figure 5: Sample ambiguous misclassifications—when visual cues do not suffice for reliable color assignment, and ground-truth labels (allocated by vehicle registration, not visual evidence) are not visually verifiable.

This error analysis delineates the true operational ceiling for VCR in surveillance: roughly 8–10pp above the current best model, with the remaining gap predominantly attributable to visual ambiguities rather than model capacity.

Implications and Future Evolution

The study rigorously demonstrates that strong synthetic augmentation—especially domain-congruent and context-preserving—combined with balanced training objectives, robust visual representations, and cautious color handling, is indispensable for class-balanced VCR in surveillance. However, it also highlights the fundamental visual limitations of color-based vehicle search, with real-world ambiguities setting a stringent upper bound even for well-calibrated ensembles.

For the research community, the implication is clear: gains will plateau without integration of rich multimodal cues (e.g., joint vehicle color, type, make, and model recognition) and uncertainty-aware inferencing—in particular, protocols that explicitly express label uncertainty and implement human-in-the-loop fallback mechanisms for operational deployment in critical law enforcement and security applications.

Conclusion

This study presents the most thorough evaluation to date of the practical limits of vehicle color recognition in realistic, long-tailed surveillance settings. It establishes Gemini-based, quality-controlled synthetic augmentation as key for unlocking minority-class accuracy, and delivers the highest reported macro-accuracy on UFPR-VeSV. Yet, crucially, it empirically demonstrates that a substantial portion of misclassifications are visual-epistemic rather than algorithmic, setting clear expectations for future work—emphasizing multi-task modeling, uncertainty quantification, and interactive recognition protocols for robust, real-world deployment.