UniPAR: A Unified Framework for Pedestrian Attribute Recognition

Abstract: Pedestrian Attribute Recognition is a foundational computer vision task that provides essential support for downstream applications, including person retrieval in video surveillance and intelligent retail analytics. However, existing research is frequently constrained by the ``one-model-per-dataset" paradigm and struggles to handle significant discrepancies across domains in terms of modalities, attribute definitions, and environmental scenarios. To address these challenges, we propose UniPAR, a unified Transformer-based framework for PAR. By incorporating a unified data scheduling strategy and a dynamic classification head, UniPAR enables a single model to simultaneously process diverse datasets from heterogeneous modalities, including RGB images, video sequences, and event streams. We also introduce an innovative phased fusion encoder that explicitly aligns visual features with textual attribute queries through a late deep fusion strategy. Experimental results on the widely used benchmark datasets, including MSP60K, DukeMTMC, and EventPAR, demonstrate that UniPAR achieves performance comparable to specialized SOTA methods. Furthermore, multi-dataset joint training significantly enhances the model's cross-domain generalization and recognition robustness in extreme environments characterized by low light and motion blur. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR

• The paper presents a unified Transformer approach that fuses multi-modal visual data with late semantic alignment for precise pedestrian attribute recognition.
• The methodology uses a phased fusion encoder and dynamic classification head to enhance joint training, yielding notable mA and F1 improvements on diverse benchmarks.
• Experimental results indicate superior cross-domain generalization and validate the effectiveness of modality-agnostic backbones and dataset-specific routing.

UniPAR: A Unified Transformer Framework for Pedestrian Attribute Recognition

Motivation and Problem Definition

Pedestrian Attribute Recognition (PAR) underpins numerous high-level computer vision applications, such as surveillance-based person retrieval and retail analytics, by classifying semantic attributes (e.g., gender, apparel, accessories) from visual data. Despite rapid progress driven by deep learning, existing approaches predominantly operate under a "one-model-per-dataset" paradigm, leading to poor scalability and weak generalization across domains, modalities, and attribute taxonomies. Notably, SOTA models struggle to transition their performance to real-world, cross-domain settings, due in part to benchmark-centric design and overfitting to specific dataset characteristics.

Figure 1: Comparative schematic: Existing PAR models exhibit per-dataset specialization, whereas UniPAR enables unified multi-domain processing with late-stage multi-modal semantic alignment.

UnifiedPAR: Technical Architecture

To overcome the prevailing domain-specific limitations, UniPAR introduces a unified, Transformer-based framework constructed for joint, multi-modal, multi-domain pedestrian attribute recognition. The core architectural strategy comprises a modality-agnostic visual backbone, a phased fusion encoder, and a dynamic classification head.

Figure 2: The overall architecture of the UnifiedPAR framework, highlighting the phased fusion encoder and modality-agnostic design.

Multi-modal Visual Embedding and Preprocessing

UniPAR deploys modality-specialized convolutional stems to produce initial patch embeddings for divergent modalities, including RGB, video, and event data. Each embedded sequence is enhanced by integrating spatial, temporal (for sequence inputs), and modality-type positional encodings. A time adapter module compresses multi-frame information efficiently, preserving salient temporal dynamics while standardizing representations.

Phased Fusion Encoder: Late Deep Visual-Semantic Alignment

Distinct from prior Transformer-based designs, UniPAR implements a two-stage (phased) ViT backbone. Visual tokens traverse $L-1$ Transformer layers focused exclusively on extracting comprehensive, global, and local visual context. Only at the final ($L$-th) layer are attribute query tokens (derived from semantic embeddings of attribute definitions) concatenated with these visual features. This late fusion paradigm ensures robust, modality-agnostic visual feature extraction prior to semantic alignment. The final layer’s self-attention mechanism then explicitly aligns each query with relevant visual evidence, facilitating precise attribute localization even under heterogeneous input conditions.

Dynamic Classification Head and Unified Data Scheduling

To reconcile diverse attribute spaces across datasets, UniPAR defines dataset-specific linear classifiers with batch normalization mapped dynamically based on the current input’s attribute definition. This approach avoids parameter interference and maintains classification granularity without bespoke head engineering for each scenario. A novel data scheduling mechanism standardizes input collation and asynchronizes source-batch formation and training iterations, ensuring stable joint training and avoiding domain inter-batch contamination.

Experimental Evaluation and Results

UniPAR is extensively validated across MSP60K (multi-domain, degraded synthetic images), DukeMTMC-Attribute (video, multi-camera), and EventPAR (large-scale event-based images with expanded attribute sets) benchmarks. Experiments rigorously consider joint versus individual training regimes, ablate key architectural choices, and analyze loss-weighting strategies.

Key Performance Findings:

• Joint Training Efficacy: Across all three benchmarks, joint training consistently improves mean Accuracy (mA) and F1 compared to individually trained baselines, with notable gains in domain generalization and motion/lighting robustness.
• Comparison to SOTA: On MSP60K, UniPAR achieves mA = 75.12% and F1 = 85.15%, outperforming previous CNN- and Transformer-based methods, except LLM-augmented models that require significantly higher computational overheads. On EventPAR, UniPAR attains mA = 86.90% and F1 = 87.53%, confirming its superior multi-modality robustness.
• Ablation Analysis: The phased semantic fusion and data-scheduling mechanisms are directly linked to cross-domain transfer and stability. Semantic query encoding via dataset-tuned representations outperforms generic (e.g., CLIP, BERT) text encoders, especially on smaller or more heterogeneous datasets.

  Figure 3: Visual qualitative analysis of generalization: UniPAR detects most attributes correctly across modalities, with error markers denoting failures in challenging conditions.

  

  Figure 4: Joint training improves attribute localization and reduces errors under cross-domain evaluation, compared to individually trained models.

Limitations and Future Directions

Despite robust joint training performance, UniPAR’s effectiveness on pure RGB benchmarks is somewhat limited compared to its results with event-based or multimodal input, highlighting the need for further enhancement of single-modal backbones. Additionally, while the model’s classification head generalizes across datasets, a shift toward open-vocabulary attribute prediction is essential for adapting to novel domains or evolving attribute ontologies. Future research should investigate incorporation of further modalities (e.g., IR, depth) and refinement of semantic querying for scalable, universal visual understanding frameworks.

Conclusion

UniPAR represents a significant advancement in unified PAR modeling by consolidating diverse modalities, domains, and attribute sets into one end-to-end trainable Transformer-based system. Late-stage, deep cross-modal fusion and dataset-specific dynamic routing serve as the keystones for its superior generalizability and efficiency. The practical implications extend to robust surveillance and retail analytics in unconstrained environments, while theoretically, UniPAR paves the way for foundation models linking language, vision, and dynamic sensorial domains. Future extensions involving open-vocabulary reasoning and broader sensor inclusion offer promising directions for the development of generalizable human-centric AI systems.

(2603.05114)