Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes

Abstract: Multi-view crowd tracking estimates each person's tracking trajectories on the ground of the scene. Recent research works mainly rely on CNNs-based multi-view crowd tracking architectures, and most of them are evaluated and compared on relatively small datasets, such as Wildtrack and MultiviewX. Since these two datasets are collected in small scenes and only contain tens of frames in the evaluation stage, it is difficult for the current methods to be applied to real-world applications where scene size and occlusion are more complicated. In this paper, we propose a Transformer-based multi-view crowd tracking model, \textit{MVTrackTrans}, which adopts interactions between camera views and the ground plane for enhanced multi-view tracking performance. Besides, for better evaluation, we collect and label two large real-world multi-view tracking datasets, MVCrowdTrack and CityTrack, which contain a much larger scene size over a longer time period. Compared with existing methods on the two large and new datasets, the proposed MVTrackTrans model achieves better performance, demonstrating the advantages of the model design in dealing with large scenes. We believe the proposed datasets and model will push the frontiers of the task to more practical scenarios, and the datasets and code are available at: https://github.com/zqyq/MVTrackTrans.

• The paper introduces MVTrackTrans, a transformer-based framework that integrates view-ground interactions for enhanced multi-view crowd tracking in complex, large-scale scenes.
• It utilizes a shared ResNet backbone, deformable transformer encoder, and dense heatmap regression to effectively fuse spatial-temporal features despite occlusions.
• Experimental results on MVCrowdTrack and CityTrack datasets show significant improvements in MOTA and IDF1 metrics compared to previous CNN-based approaches.

Multi-view Crowd Tracking Transformer with View-Ground Interactions: Advancing Large-Scale Multi-View MOT

Introduction: Challenges in Multi-view Crowd Tracking

Multi-view crowd tracking in real-world environments presents significant complexity due to large spatial coverage, high crowd density, frequent occlusions, and extended temporal durations. Existing benchmarks, primarily Wildtrack and MultiviewX, are constrained to limited scene size and short sequences, impeding generalization to practical applications. Previous state-of-the-art (SOTA) methods rely predominantly on CNN-based models and fail to extensively explore architectures capable of learning robust spatial and temporal feature associations across multiple views. This paper introduces MVTrackTrans, a transformer-based multi-view tracking method with explicit view-ground interaction, deployed and evaluated on two large-scale datasets (MVCrowdTrack and CityTrack) purposely constructed for such challenging scenarios (2604.19318).

Figure 1: Comparison of the proposed MVCrowdTrack and CityTrack datasets with MultiviewX and Wildtrack, highlighting scene coverage, average track lengths, and crowd scales.

Datasets: MVCrowdTrack and CityTrack

The MVCrowdTrack and CityTrack datasets provide substantial advancements over prior benchmarks. MVCrowdTrack is captured on a campus environment, spans $120\,\text{m} \times 80\,\text{m}$, utilizes 7 synchronized cameras, and covers 4,122 frames with average track lengths of 176 frames. CityTrack, based on the CityStreet dataset, is resampled and annotated for multi-view tracking at 4 fps with 2,588 frames, averaging 228 track frames per identity. Both datasets feature denser temporal annotations, larger spatial coverage, longer trajectories, and more consistent IDs compared to Wildtrack and MultiviewX. These characteristics establish more stringent benchmarks for multi-view tracking evaluation in realistic environments.

Figure 2: Multi-camera views and ground-plane layout in MVCrowdTrack, illustrating calibration and coverage.

MVTrackTrans Architecture

Feature Extraction and Multi-view Fusion

MVTrackTrans employs a shared ResNet backbone to extract multi-level features from all camera views. These are projected onto the ground plane using calibrated camera parameters, aggregated via multi-height bilinear sampling, and fused to obtain the BEV representation.

Transformer-based Multi-view Tracking Encoding

A deformable Transformer encoder processes fused ground features from consecutive frames. Track queries are sampled from previous frame ground features at verified detection locations. The View-Ground Interaction module fuses track queries and view-specific queries via cross-attention, ensuring robust integration of spatial-temporal information from multiple camera perspectives and ground-plane representations.

Figure 3: MVTrackTrans pipeline: feature extraction/fusion, tracking encoding, view-ground interaction, and decoding.

Figure 4: View-ground interaction module details, enabling fusion via cross-attention between track and camera-view queries.

Multi-view Tracking Decoding and Supervision

MVTrackTrans utilizes parallel branches:

• Offset Decoder applies MSDA attention to propagate and refine track queries (motion offsets).
• Heatmap Decoder regresses a dense location heatmap over the ground plane. Losses are weighted using an uncertainty-driven strategy, balancing detection and temporal association for enhanced training stability.

Experimental Results

MVTrackTrans achieves superior performance on both MVCrowdTrack and CityTrack, consistently outperforming established baselines (EarlyBird, MVFlow, TrackTacular) in MOTA, IDF1, and MT metrics. On MVCrowdTrack, MVTrackTrans achieves 63.87 MOTA and 59.06 IDF1, surpassing TrackTacular by over 1 point in both metrics and outperforming EarlyBird by over 9 MOTA and 5 IDF1 points. On CityTrack, MVTrackTrans similarly sets new benchmarks (55.39 MOTA, 34.41 IDF1). The method demonstrates robust identity preservation and detection accuracy across long-duration sequences and severe occlusions.

Ablation studies highlight the necessity of view-ground interaction modules and dense heatmap regression. Cross-attention fusion outperforms self-attention, and dense heatmap regression loss yields significant gains over direct coordinate regression, confirming the importance of dense spatial supervision and effective multi-view fusion strategies.

Figure 5: Predicted trajectory visualizations – MVTrackTrans tracks more identities for longer periods compared to other methods.

Implications and Outlook

MVTrackTrans demonstrates the efficacy of transformer architectures for multi-view crowd tracking, where spatial and temporal associations are central bottlenecks for SOTA CNN-based models. Explicit view-ground cross-attention significantly strengthens identity retention under occlusion and improves matching robustness in dynamic environments. The large-scale MVCrowdTrack and CityTrack datasets redefine evaluation criteria; practical deployment demands handling thousands of frames and high-density crowds, and MVTrackTrans is validated against these requirements.

From a theoretical standpoint, this research highlights that transformers, when appropriately engineered for multi-view fusion and temporal modeling, are primed to address the limitations of prior CNN-based crowd tracking. The explicit query-based framework and attention mechanisms can be generalized for other multi-sensor, multi-modal tracking applications, including vehicle tracking, surveillance, and smart transportation systems.

Future directions include enhancing scalability for city-scale video analytics, adapting to variable camera layouts, leveraging additional modalities (depth, thermal), and integrating look-ahead prediction for real-time applications. Transformer-based architectures with adaptive attention fusion are likely to underpin future advances in robust tracking across crowded, occlusion-heavy domains.

Conclusion

MVTrackTrans, the transformer-based multi-view crowd tracking framework, achieves consistent superiority on large-scale real-world datasets through sophisticated view-ground interaction and robust spatial-temporal encoding. By combining dense ground-plane supervision, cross-view fusion, and transformer query mechanisms, MVTrackTrans advances the frontier of crowd tracking. The MVCrowdTrack and CityTrack datasets further facilitate evaluation in complex scenarios, marking a critical step toward practical, scalable multi-view crowd tracking systems (2604.19318).