Spatial-Temporal Expert Learning for Video-based Person Re-identification

Abstract: Video-based person re-identification (Re-ID) aims to retrieve the same identity in the query video clips from the gallery video clips. To solve this problem, exploiting fine-grained features is of great importance, especially when discriminating identities that are similar in appearance. In this paper, we propose to enhance the ability to explore fine-grained information with a novel input-aware extendable expert module. Instead of updating the network parameters with every sample in the dataset, we aim to train the experts within specific subsets that only contain similar samples and promote their ability to exploit fine-grained information within these similar samples. To achieve this goal, we incorporate two mechanisms in this module: input-aware expert selection mechanism and spatial-temporal selection mechanism. The first mechanism dynamically activates a set of experts on subsets of similar samples, pushing the experts to exploit subtle differences between these similar samples, while the second one further increases their sensitivity to the fine-grained differences in spatial and temporal aspects and allows the experts to dynamically utilize them for different input samples. In addition, to facilitate the expert module, we design an extendable scheme that allows the module to flexibly add new experts when necessary. As a result, our method achieves outstanding performance on two large-scale datasets.

• The paper proposes an input-aware expert module that dynamically selects specialized branches to capture fine-grained spatial and temporal features.
• The method achieves high accuracy with mAP up to 87.0% and rank-1 rates over 91.6% on benchmark datasets MARS and LS-VID.
• The design enables adaptive expert extension and diversity-driven optimization, offering robust and scalable solutions for video surveillance.

Spatial-Temporal Expert Learning for Fine-Grained Video Person Re-Identification

Introduction and Motivation

Video-based person re-identification (Re-ID) remains a critical issue in multi-camera surveillance and smart city scenarios, especially under challenging conditions where different individuals exhibit extremely similar visual appearances. Recent literature has investigated attention-based and part-level approaches to enhance fine-grained feature extraction, but conventional methods tend to bias representation learning towards globally discriminative, dataset-wide patterns, thereby neglecting subtle cues—such as minor clothing details or gait patterns—that are only present in specific subsets of samples. This paper proposes an input-aware, extendable expert module aimed at explicitly specializing feature extraction for these challenging subsets, with an emphasis on spatial-temporal discrimination that adapts dynamically to both appearance and motion cues.

Methodology

Input-Aware Extendable Expert Module

The cornerstone of the method is the expert module, which consists of several expert layers, each containing multiple expert branches and a dynamic selection mechanism. For a given input, each expert layer evaluates the relevance scores produced by the experts via learned parametric mappings. The most relevant expert is activated for each input based on these scores, while the others are deactivated. This ensures that each expert becomes responsible for a narrow subset of visually similar samples, thereby promoting specialization to fine-grained discriminative cues and avoiding the dilution of representations common in global optimization over the entire dataset.

Figure 1: Architecture of the proposed expert module. Each input is dynamically routed to the most relevant expert in each layer, with specialization enforced via a selection mechanism and the ability to extend the number of experts as needed.

The extendable design introduces a wait-list expert in every layer. If the incoming sample is not well-matched to any existing expert, the wait-list expert is activated and subsequently appended to the layer, allowing the system to scale expert capacity adaptively during training. This eliminates the need for hand-tuning the expert pool cardinality, which can be dataset- and scenario-dependent.

Spatial-Temporal Selection Mechanism

Each activated expert further splits its representation channels via an input-aware spatial-temporal selection mechanism. A significance vector is computed for each channel to determine whether it is best processed by a spatial branch (focusing on intra-frame variations such as appearance or attributes) or a temporal branch (focusing on inter-frame dynamics like motion and gait). The mapping of channels to branches is performed via binary selection driven by improved Semhash, and the final feature is an additive fusion of the outputs of both branches.

This parallel specialization forces every channel to focus on either spatial or temporal information as needed for each input, thus maximizing the utilization of fine-grained clues in both domains.

Diversity-Driven Optimization

To further ensure orthogonal specialization among experts, a diversity loss is introduced. This loss penalizes the cosine similarity between the parameter vectors of the spatial and temporal branches across experts within the same layer. The total loss combines cross-entropy, batch-wise triplet, and diversity losses, ensuring both identification accuracy and expert specialization.

Experimental Evaluation

The expert module is inserted after a ResNet-50 backbone and is evaluated on the MARS and LS-VID datasets, two of the largest and most competitive video Re-ID benchmarks. Experiments include state-of-the-art method comparison, ablations, and detailed visualizations.

On the MARS dataset, the method achieves 87.0% mAP and 91.6% rank-1, and on LS-VID, 81.0% mAP and 88.3% rank-1, outperforming or matching prior fine-grained and spatiotemporal Re-ID approaches. Notably, the fully dynamic expert module with three layers and adaptively extended expert cardinality achieves the best results without exhaustive manual tuning.

Ablations indicate that:

• Disabling the expert selection mechanism (e.g., using random or average outputs) causes significant drops in performance, validating the necessity of dynamic specialization.
• Removing the spatial-temporal selection or replacing it with simpler fusions erodes discriminative power, highlighting the importance of dynamic spatial/temporal channel allocation.
• Incorporating the diversity loss and adaptive expert number yields measurable gains over static configurations.

  Figure 2: Visualization of input samples, feature activations, and expert selection. Samples with similar appearance activate similar experts; output features after expert processing are more discriminative.

Theoretical and Practical Implications

From a theoretical perspective, the methodology provides evidence that automatic specialization via dynamic routing and extendable expert pools can overcome the generalization/fine-granularity tradeoff endemic to conventional identification architectures. The capacity to assign local experts to subclasses of hard samples aligns with principles from mixture-of-experts models but is extended here to a spatial-temporal, input-adaptive and scalable setting.

Practically, this approach is suitable for large-scale, heterogeneous video surveillance deployments. Its input-aware and extendable nature allows it to flexibly handle new data domains or shifts in sample distributions, which are prevalent in real-world surveillance due to environmental, seasonal, or demographic factors. The modularity of the expert module facilitates easy integration with privacy-preserving mechanisms such as federated learning, and the design is naturally compatible with edge-deployable architectures, as only the relevant experts require activation and updating.

Speculation on Future Developments

Dynamic specialization as implemented here is likely to find broader application in other fine-grained visual domains, such as action recognition, vehicle Re-ID, and animal tracking. Further work could explore cross-modal experts, multi-step routing, or tighter integration with transformer-based architectures to amplify both spatial-temporal modeling capacity and expert diversity. Investigation into self-supervised or unsupervised variants could make such models applicable where labels are scarce, and ongoing research into model compression could allow deployment in highly resource-constrained IoT scenarios.

Conclusion

This paper introduces an input-aware, extendable expert module with dual spatial-temporal specialization for video-based person Re-ID. The approach demonstrates measurable improvements over strong baselines and provides a scalable framework for fine-grained, adaptive discrimination. The implications for adaptive visual representation learning in dynamic, real-world environments are significant, with further research opportunities in enhanced specialization, self-supervision, and responsible deployment.