VehicleMAE-V2: Pre-trained Vehicle Model

• The paper introduces VehicleMAE-V2, which integrates symmetry-guided, contour-guided, and semantics-guided modules into a masked auto-encoder to improve vehicle perception.
• The model employs a ViT-Base/16 encoder with a transformer decoder and leverages multimodal pre-training using a 4M-vehicle dataset to boost downstream task accuracy.
• Evaluations on tasks like attribute recognition, detection, re-identification, and fine-grained classification demonstrate significant improvements over traditional MAE frameworks.

VehicleMAE-V2 is a vehicle-centric pre-trained large model designed to address deficiencies in typical masked auto-encoder (MAE) pipelines when modeling generalizable representations for vehicle perception. By incorporating three structured vehicle priors—symmetry, contour, and semantics—during multimodal pre-training, VehicleMAE-V2 achieves superior performance across a broad spectrum of downstream vehicle-centric tasks, including attribute recognition, detection, re-identification, fine-grained classification, and part segmentation. This is accomplished through the integration of the Symmetry-guided Mask Module (SMM), Contour-guided Representation Module (CRM), and Semantics-guided Representation Module (SRM), each separately enforcing domain-specific knowledge during masked token reconstruction. Pre-training utilizes the Autobot4M dataset, comprising approximately 4 million vehicle crops and 12,693 textual model descriptions (Wu et al., 22 Dec 2025).

1. Model Architecture and Pre-training Objective

VehicleMAE-V2 extends the standard MAE framework by introducing structured priors at multiple stages of token masking and reconstruction:

• Input images $I \in \mathbb R^{224 \times 224 \times 3}$ are partitioned into $14 \times 14 = 196$ non-overlapping $16 \times 16$ patches.
• 75% of patches are masked via SMM; unmasked patches are projected to 768-dimensional embeddings, prepended with a learnable [CLS] token, and augmented by positional encodings.
• The transformed tensor is input to a 12-layer ViT encoder; outputs are projected to 512 dimensions, recombined with learnable masked tokens, and re-encoded using a secondary positional encoding before reconstruction by an 8-layer Transformer decoder.

The full pre-training loss is a weighted sum of pixel-level and structured prior objectives:

$$L = \lambda_r L_r + \lambda_{mim} L_{mim} + \lambda_{cls} L_{cls} + \lambda_{cf} L_{cf} + \lambda_{cs} L_{cs} + \lambda_{vt} L_{vt}$$

with coefficients $\lambda_r=4$, $\lambda_{mim}=0.02$, $\lambda_{cls}=0.02$, $\lambda_{cf}=\lambda_{cs}=\lambda_{vt}=1$ in the best-performing variant.

2. Symmetry-guided Mask Module (SMM)

SMM exploits the inherent approximate left–right symmetry found in vehicle instances. For each patch $i$ in the detected bounding box set $\mathcal B$ (determined by the YAEN angle detector with estimated vehicle yaw $\theta$), a symmetric partner $i'$ is computed by geometric reflection. The enforced rule is that at least one patch in every symmetric pair $(i,i')$ is always masked:

$$\text{MaskSet} \leftarrow \{\,i\,|\, U(0,1) < M_R\}$$

with post-processing correction ensuring for all $(i,i')$:

$$\max\left[\mathbf{1}_{\{i \in \text{MaskSet}\}}, \mathbf{1}_{\{i' \in \text{MaskSet}\}}\right] \geq 1$$

This strategy reduces information redundancy and selects high-informative regions, improving latent representation in high mask ratio regimes (default $M_R = 0.75$).

3. Contour-guided Representation Module (CRM)

CRM seeks to enforce global shape preservation beyond what pixel-level MSE ($L_r$) achieves. The BDCN edge detector extracts contour maps; their features $F^s \in \mathbb R^{197 \times 768}$ (197 = 196 patches + 1 CLS token) are compared to decoder outputs $F^t \in \mathbb R^{197 \times 512}$ as follows:

• Patch-level distribution alignment: Patch vectors are projected to $K$-way probabilities (MLPs $\theta', \theta$), using cross-entropy

$$L_{mim} = -\sum_{i=1}^{N_p} P^{\text{patch}_{\theta'}}(F^s_i)^\top \log P^{\text{patch}_\theta}(F^t_i)$$

• Class-token distribution alignment: Same form for CLS token,

$$L_{cls} = -P^{\text{cls}_{\theta'}}(F^s_\text{cls})^\top \log P^{\text{cls}_\theta}(F^t_\text{cls})$$

These alignment losses facilitate holistic structure preservation during masked reconstruction.

4. Semantics-guided Representation Module (SRM)

SRM injects semantic vehicle knowledge via cross-modal alignment with frozen CLIP models:

• Cross-modal distillation ($L_{cf}$): Enforces L2 distance between normalized CLIP visual embedding $V^c$ and decoder output $F^t$

$$L_{cf} = \left\|\frac{F^t}{\|F^t\|_2} - \frac{V^c}{\|V^c\|_2}\right\|_2^2$$

• Similarity-distribution consistency ($L_{cs}$): Attribute texts pool $W = \{w_1, \dots, w_m\}$, similarity computed via softmax

$$s_j(\tilde x, \tilde w_j) = \frac{\exp((\tilde x \cdot \tilde w_j)/\tau)}{\sum_{n=1}^m \exp((\tilde x \cdot \tilde w_n)/\tau)}$$

KL divergence regularizes cross-modal consistency:

$$L_{cs} = \mathrm{KL}[S(\tilde V^c, W) \,\|\, S(\tilde F^t, W)] + H(S(\tilde F^t, W))$$

• Vision–text contrastive ($L_{vt}$): Paired prompts (captions with box coordinates, size ratio, yaw angle) encoded via CLIP, enforcing cosine similarity:

$$L_{vt} = \frac{1}{N_b} \sum_{i=1}^{N_b} \mathrm{CosineEmbedding} \left( \frac{\bar P_i}{\|\bar P_i\|_2}, \frac{F^w_i}{\|F^w_i\|_2} \right)$$

SRM reduces feature confusion in semantic discrimination and enables robust cross-modal representation learning.

5. Data Foundation: The Autobot4M Dataset

VehicleMAE-V2 training leverages Autobot4M, comprising 4,020,368 tightly cropped vehicle images sourced from Autobot1M, the PKU-VD train split, and SODA10M (YOLOv5-cropped). The accompanying 12,693 textual vehicle-model descriptions (with 11 attributes each) serve as language modality. Preprocessing pipelines incorporate YOLOv5 (bounding boxes), BDCN (edges), YAEN (angle estimation), and ChatGPT-generated semantic captions.

Source Dataset	Images (Millions)	Text Descriptions
Autobot1M	1.03	12,693
PKU-VD	1.89	12,693
SODA10M	1.11	12,693

This multimodal curation facilitates the structured pre-training required for VehicleMAE-V2.

6. Training Configuration and Complexity

VehicleMAE-V2 utilizes a ViT-Base/16 encoder (12 layers, 768-d embedding), an 8-layer Transformer decoder, and CLIP-ViT-B/16 references. Model complexity is 122.41 million parameters and 10.98 GFLOPs. Training employs AdamW optimization (LR = $2 \times 10^{-4}$; weight decay = 0.04; batch size = 512) for 100 epochs. Pre-training variants include use of Autobot1M or Autobot4M datasets; no specialized learning rate schedule is reported.

7. Task Evaluation, Performance, and Ablation

Performance is measured across five fine-tuning benchmarks with consistent metrics gains over prior approaches (ImageNet-MAE, DINO, IBOT, and VehicleMAE baseline):

Task	VehicleMAE-V2 (Autobot4M) Metrics
Attribute Recognition	mA: 93.19, Acc: 95.91, Prec: 96.89, Rec: 97.28, F1: 96.98
Object Detection (VFM-Det)	AP$_{[50:95]}$: 48.5, AP$_{50}$: 67.4, AP$_{75}$: 53.8
Re-ID (TransREID, VeRi)	mAP: 87.3, Rank-1: 98.0
Fine-grained Recognition (TransFG, Stanford Cars)	Acc: 94.9
Part Segmentation (SETR, PartImageNet)	mIoU: 75.04, mAcc: 81.04

Ablation studies reveal that the inclusion of CRM alone yields +7–8 mAP in re-ID, SRM losses further boost by +1–2 mAP, and SMM outperforms random masks, especially at high masking ratios. Increasing training data size (Autobot1M→Autobot4M) and backbone capacity (ViT-Large/16) yield additional improvements, notably in re-ID and attribute recognition.

8. Domain-Specific Insights, Limitations, and Prospects

VehicleMAE-V2 demonstrates:

• Structured prior impact: SMM reduces redundancy and enables robust representation under high masking. CRM enforces shape-aware reconstructions, improving spatially sensitive tasks. SRM injects cross-modal semantic richness, critical for fine-grained and identity tasks.
• Scaling effects: Expansion of data volume and backbone scale are positively correlated with downstream gains, especially for re-ID and semantic tasks.
• Instance-centric operational limitations: Application in full-scene detection requires external vehicle detection (e.g., VFM-Det) as pre-training remains crop-focused.
• Multimodal extensibility: The framework currently leverages only RGB and text; future directions propose incorporating modalities like LiDAR, infrared, depth, and event camera data for comprehensive vehicle foundation modeling.

A plausible implication is that the integration of structured multimodal priors during MAE-style pre-training, as instantiated in VehicleMAE-V2, sets a precedent for domain specialization beyond general-purpose foundation models, with ramifications for instance-level perception and multimodal fusion in intelligent transportation and surveillance systems (Wu et al., 22 Dec 2025).