Temporal Prototyping and Hierarchical Alignment for Unsupervised Video-based Visible-Infrared Person Re-Identification

Abstract: Visible-infrared person re-identification (VI-ReID) enables cross-modality identity matching for all-day surveillance, yet existing methods predominantly focus on the image level or rely heavily on costly identity annotations. While video-based VI-ReID has recently emerged to exploit temporal dynamics for improved robustness, existing studies remain limited to supervised settings. Crucially, the unsupervised video VI-ReID problem, where models must learn from RGB and infrared tracklets without identity labels, remains largely unexplored despite its practical importance in real-world deployment. To bridge this gap, we propose HiTPro (Hierarchical Temporal Prototyping), a prototype-driven framework without explicit hard pseudo-label assignment for unsupervised video-based VI-ReID. HiTPro begins with an efficient Temporal-aware Feature Encoder that first extracts discriminative frame-level features and then aggregates them into a robust tracklet-level representation. Building upon these features, HiTPro first constructs reliable intra-camera prototypes via Intra-Camera Tracklet Prototyping by aggregating features from temporally partitioned sub-tracklets. Through Hierarchical Cross-Prototype Alignment, we perform a two-stage positive mining process: progressing from within-modality associations to cross-modality matching, enhanced by Dynamic Threshold Strategy and Soft Weight Assignment. Finally, {Hierarchical Contrastive Learning} progressively optimizes feature-prototype alignment across three levels: intra-camera discrimination, cross-camera same-modality consistency, and cross-modality invariance. Extensive experiments on HITSZ-VCM and BUPTCampus demonstrate that HiTPro achieves state-of-the-art performance under fully unsupervised settings, significantly outperforming adapted baselines and establishes a strong baseline for future research.

• The paper introduces HiTPro, a clustering-free framework that combines temporal feature encoding, intra-camera tracklet prototyping, and hierarchical cross-prototype alignment for unsupervised video-based VI-ReID.
• HiTPro achieves 56.58% Rank-1 and 44.27% mAP on HITSZ-VCM, outperforming adapted unsupervised baselines by up to 12.10 Rank-1 points and 8.04 mAP points.
• The results show that staged intra-modality alignment must precede cross-modality learning, while dynamic thresholds, soft weighting, and temporal scheduling improve the reliability of automatically mined positives.

Problem formulation and motivation

Visible-infrared person re-identification (VI-ReID) matches pedestrian identities across RGB and infrared (IR) cameras, but the vast majority of existing work operates at the image level or assumes full identity supervision. This paper addresses the unsupervised video-based VI-ReID setting: learning cross-modality, cross-camera identity associations from RGB and IR tracklets without any identity labels. The authors argue that this setting cannot be solved by porting image-level unsupervised VI-ReID pipelines, for three reasons: the number of tracklets is far smaller than the number of frames, violating the density assumptions underlying DBSCAN-based clustering; tracklet representations exhibit large intra-identity variation due to pose dynamics, motion blur, and background clutter; and each identity must be matched across two orthogonal heterogeneity sources simultaneously—cross-camera shift and cross-modality gap. The paper also exploits a structural prior specific to surveillance video: tracklets observed by a single camera within a short time window are mutually identity-disjoint, providing label-free discrimination signals.

The HiTPro framework

HiTPro is a prototype-driven, clustering-free framework composed of four components.

Temporal-aware Feature Encoder (TFE). TFE builds on the AGW backbone with modality-specific shallow layers and a shared deep trunk. A two-layer Transformer temporal encoder models inter-frame dependencies, followed by an Adaptive Frame-weighting Module (AFM) that computes softmax attention weights over frames and produces a weighted video-level embedding. This suppresses occluded or noisy frames and yields compact tracklet representations.

Intra-Camera Tracklet Prototyping (ICTP). Each tracklet is partitioned into $K$ non-overlapping sub-tracklets ($K=4$ by default), each encoded by TFE; the tracklet prototype is the mean of sub-tracklet features. Because intra-camera tracklets are identity-disjoint, each prototype serves as a reliable pseudo-center without any cross-camera association or clustering, sidestepping the sparsity problems that break density-based clustering on tracklets.

Hierarchical Cross-Prototype Alignment (HCPA). Positive mining proceeds in two stages: first, intra-modality cross-camera mining, where for each prototype the per-camera best match is retained if it exceeds a similarity threshold; second, cross-modality mining, matching visible prototypes to IR prototypes and vice versa. Two refinements improve reliability: a Dynamic Threshold Strategy (DTS) sets the threshold per instance as a fraction of its current maximum match confidence, with the fraction linearly decaying from 0.99 to 0.90 over training; and Soft Weight Assignment (SWA) reweights retained positives via a temperature-scaled softmax over their similarities.

Hierarchical Contrastive Learning (HCL). Three InfoNCE-style losses align sub-tracklet features with prototypes progressively: an intra-camera loss, an intra-modality cross-camera loss, and a cross-modality loss. A Hierarchical Loss Scheduling (HLS) scheme activates the cross-camera loss at epoch 5 and the cross-modality loss at epoch 15, avoiding premature exposure to noisy cross-modal matches. Prototypes are updated by exponential moving average, with each sub-tracklet feature also updating its reliably associated positive prototypes.

Empirical results

Experiments use HITSZ-VCM (500 training identities, 1,481 visible and 1,445 IR tracklets after label removal) and BUPTCampus (1,000 training identities, 2,335 tracklets per modality). Because no prior unsupervised cross-modal video methods exist, the authors adapt image-level unsupervised VI-ReID methods (e.g., PGM, SDCL) and single-modality unsupervised video methods (e.g., UTAL, UFCS, DRMPCL) as baselines.

Method	Type	HITSZ-VCM I→V R1 / mAP	HITSZ-VCM V→I R1 / mAP
DRMPCL	Unsup. video	44.48 / 34.23	45.12 / 37.31
UFCS	Unsup. video	42.98 / 33.90	44.57 / 35.20
SDCL	Unsup. image	20.35 / 19.21	21.43 / 18.74
HiTPro	Unsup. video	56.58 / 44.27	57.98 / 45.70

On HITSZ-VCM, HiTPro exceeds the second-best unsupervised method by 12.10 points Rank-1 and 8.04 points mAP under infrared-to-visible, and notably surpasses several fully supervised image-based methods (e.g., CAJ at 56.59 R1 is essentially tied, while DDAG at 54.62 is exceeded). On BUPTCampus, the margins are larger: 42.91 R1 / 41.78 mAP (I→V), a 19.03-point Rank-1 and 16.66-point mAP improvement over the runner-up, again exceeding some supervised image-based baselines. Absolute performance is lower on BUPTCampus, which the authors attribute to its larger cross-modality gap and more dynamic scenes.

Ablations on HITSZ-VCM attribute the gains as follows. The clustering-based baseline achieves only 10.73 R1 (I→V); adding TFE or ICTP individually raises this to roughly 20 R1, and their combination to 31.62. Adding HCPA and HCL yields the largest single jump, to 56.58 R1 (+24.96). Within HCL, the cross-modality loss applied alone degrades performance to 15.05 R1—the paper's clearest evidence that the modality gap dominates inter-camera variation and that intra-modality alignment must precede cross-modal alignment. HLS, DTS, and SWA each contribute a further 2–3 points. Parameter studies show sensitivity to the sub-tracklet count $K$, with a consistent optimum at $K=4$–5, and to the scheduling epochs, with cross-modality alignment initiated before epoch 10 causing a noticeable accuracy drop.

Limitations and open questions

The paper concedes several limitations. A substantial gap remains relative to the strongest supervised methods—on HITSZ-VCM, SIMFGA reaches 75.55 R1 / 64.91 mAP (I→V) versus HiTPro's 56.58 / 44.27—so unsupervised performance is not yet competitive at the frontier. Performance may degrade under poor tracking quality or severe occlusion, since ICTP assumes that tracklets are correctly segmented and identity-disjoint; the framework does not handle imperfect or fragmented tracklets. The method's hyperparameters ($K$, $e_{intra}$, $e_{cross}$, threshold schedules) are tuned per dataset, and the paper does not establish whether these transfer to new deployments. Open questions include robust prototype-driven learning under noisy tracklets and extension of the paradigm to other cross-modal retrieval tasks.

Conclusion

This paper formulates unsupervised video-based VI-ReID and proposes HiTPro, a clustering-free, prototype-driven framework that replaces hard pseudo-label assignment with temporal prototyping, hierarchical two-stage positive mining with dynamic thresholds and soft weights, and progressively scheduled contrastive alignment. It establishes a strong first baseline for the task, with large margins over adapted unsupervised baselines on both benchmarks and results approaching or exceeding several supervised methods.