MVTrackTrans: Transformer Crowd Tracking

• The paper introduces a transformer-based model that lifts multi-view image features into a dense BEV to accurately track pedestrians.
• It employs a multi-scale deformable Transformer encoder and a dedicated view–ground interaction module to integrate temporal context and cross-view cues.
• Empirical results on large-scale datasets demonstrate significant gains over CNN-based methods in handling occlusions and prolonged trajectories.

MVTrackTrans is a Transformer-based model for multi-view crowd tracking that estimates each person's tracking trajectories on the ground of the scene by reasoning jointly over image-view features and a fused bird’s-eye-view ground plane. It is presented in "Multi-view Crowd Tracking Transformer with View-Ground Interactions Under Large Real-World Scenes" (Zhang et al., 21 Apr 2026). The model departs from CNNs-based multi-view crowd tracking architectures through a unified design that lifts and fuses multi-view image features into a dense BEV representation, encodes temporal context across consecutive frames with a multi-scale deformable Transformer encoder, exchanges information between track queries on the ground plane and multi-view view queries through a View–Ground Interaction module, and decodes both current-frame detection heatmaps and inter-frame motion offsets with two parallel Transformer decoder branches (Zhang et al., 21 Apr 2026).

1. Problem setting and motivation

Multi-view crowd tracking estimates each person's tracking trajectories on the ground of the scene. Recent research works mainly rely on CNNs-based multi-view crowd tracking architectures, and most of them are evaluated and compared on relatively small datasets, such as Wildtrack and MultiviewX. Since these two datasets are collected in small scenes and only contain tens of frames in the evaluation stage, it is difficult for the current methods to be applied to real-world applications where scene size and occlusion are more complicated (Zhang et al., 21 Apr 2026).

MVTrackTrans is introduced together with two large real-world multi-view tracking datasets, MVCrowdTrack and CityTrack. The stated objective is better evaluation under much larger scene size and over a longer time period. This suggests that the work is positioned not only as a model proposal but also as a benchmark-expansion effort for practical multi-camera tracking regimes characterized by large scenes, severe occlusion, and prolonged trajectories.

The method processes two consecutive timesteps, $t_0-1$ and $t_0$, from $n$ calibrated, synchronized cameras. Its formulation is explicitly ground-plane centric: detections, temporal association cues, and final trajectory estimates are all mediated through BEV representations rather than through independent per-view tracking followed by later fusion (Zhang et al., 21 Apr 2026).

2. Architectural organization

MVTrackTrans has three main stages: Feature Extraction & Multi-View Fusion, Multi-View Tracking Encoding, and Multi-View Tracking Decoding (Zhang et al., 21 Apr 2026).

In the feature extraction stage, each view image $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$ is passed through a shared ResNet-FPN backbone to extract a single-scale feature map $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$. These 2D features are then lifted into a 3D voxel grid covering the scene volume by multi-height bilinear sampling. For each voxel center $(x_n,y_n,z_n)$, the eight corner projections are defined as

$$(u_n,v_n,1)^\top = K_i\,[R_i\mid T_i]\,(x_n,y_n,z_n,1)^\top .$$

Stacking across all $n$ views and collapsing the height dimension yields a fused BEV ground feature at each scale $l=1\dots L$:

$$F_l^t=\mathrm{Conv}\Bigl(\mathrm{CollapseHeight}\bigl\{\mathrm{SampleVoxel}(\hat F_{\mathrm{view}_i}^t)\bigr\}_{i=1}^n\Bigr)\in\mathbb{R}^{H_g\times W_g\times d}.$$

In the encoding stage, temporal context is incorporated by feeding the two BEV feature sets $t_0$0 and $t_0$1 into a shared multi-scale deformable Transformer encoder in Deformable DETR style. Denoting its output as BEV queries,

$t_0$2

the model obtains a dense set of ground-plane feature vectors $t_0$3, where $t_0$4.

From $t_0$5, a fixed number $t_0$6 of track queries are sampled at the previous detection centers $t_0$7:

$t_0$8

Concurrently, for each camera view the method samples $t_0$9 view queries from the view-level features $n$0, then concatenates across views:

$n$1

All queries are first refined by small feed-forward networks and then fused by the View–Ground Interaction module (Zhang et al., 21 Apr 2026).

In the decoding stage, two parallel decoder heads consume the fused queries and the current-frame BEV queries $n$2. The offset decoder attends from the track queries at $n$3 into $n$4 using deformable cross-attention guided by the previous centers $n$5:

$n$6

$n$7

In parallel, the heatmap decoder fuses $n$8 across scales via an FPN and a small convolutional head to produce a BEV center-heatmap $n$9. The final predicted positions are obtained by combining the heatmap peaks with the offset displacements. Positional encodings follow the deformable DETR convention: each BEV query is augmented with a 2D sinusoidal encoding of its grid coordinates, and all queries carry learnable content embeddings. No extra class or object tokens are used beyond the track and view queries (Zhang et al., 21 Apr 2026).

3. View–Ground Interaction module

At the heart of MVTrackTrans is the cross-attention fusion between track queries $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$0 and view queries $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$1 (Zhang et al., 21 Apr 2026). After initial FFN refinement, the module defines

$I_i^{t}\in\mathbb{R}^{H\times W\times 3}$2

where $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$3.

The cross-attention weights are

$I_i^{t}\in\mathbb{R}^{H\times W\times 3}$4

and the fused track embeddings become

$I_i^{t}\in\mathbb{R}^{H\times W\times 3}$5

This design enables each ground-plane track query to aggregate appearance and localization cues from all camera views. The paper reports that cross-attention is superior to simple self-attention in ablation, with the stated rationale that cross-attention prevents the queries from mixing track–track and view–view features (Zhang et al., 21 Apr 2026).

A plausible implication is that the module functions as a constrained information-exchange mechanism: the ground-plane trajectory state is updated using multi-view evidence while preserving the distinction between temporal track carriers and image-conditioned view carriers. In the paper’s framing, this coupling is central to identity-consistent feature embeddings when a pedestrian is heavily occluded in some cameras.

4. Training objective and supervision

MVTrackTrans supervises both center detection and motion offsets using uncertainty-weighted losses $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$6 (Zhang et al., 21 Apr 2026).

For BEV center detection, given a ground truth heatmap $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$7 with Gaussian peaks at each true center, the BEV focal loss is

$I_i^{t}\in\mathbb{R}^{H\times W\times 3}$8

An identical focal loss $I_i^{t}\in\mathbb{R}^{H\times W\times 3}$9 is applied to predicted per-view heatmaps, providing 2D supervision in addition to BEV supervision.

For motion estimation, the offset regression loss is an $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$0 objective applied at valid center pixels:

$$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$1

The total objective is

$$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$2

The factors $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$3 and $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$4 balance the magnitude of the center versus offset losses, and $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$5 are trained jointly so that the model adaptively weighs each branch (Zhang et al., 21 Apr 2026).

The supervision strategy is tightly aligned with the decoding design. Dense BEV heatmaps supply spatially distributed localization signals, while offset regression handles inter-frame displacement. This suggests that MVTrackTrans treats multi-view tracking as a coupled detection-and-motion problem on the ground plane rather than as a pure query-regression problem.

5. Datasets and evaluation protocol

The work introduces two new datasets intended to push multi-view tracking to large, real-world scenes (Zhang et al., 21 Apr 2026).

Dataset	Description
MVCrowdTrack	Collected on a $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$6 university campus, with 7 synchronized HD cameras (5312×2988), 18 min of video at 60 fps, labeled at 4 fps for 4122 frames; 342 unique pedestrian trajectories; BEV grid 1200×800 at 0.1 m per pixel; average trajectory length 176 frames; split 3297 train and 825 test
CityTrack	Re-annotated from the CityStreet dataset, re-sampled at 4 fps and densely annotated for 2588 frames; 950 pedestrians; average length 228 frames; 3 camera views; BEV grid 640×768 at 0.1 m/pixel; split first 1948 train and last 640 test

In MVCrowdTrack, per-view bounding boxes are annotated and the foot-point of each person is projected to the BEV grid. In CityTrack, the re-annotation is described as dense after resampling from the original 500 frames at 2 fps to 4 fps (Zhang et al., 21 Apr 2026).

Evaluation is performed in BEV using standard MOT metrics with a matching threshold $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$7 for MVCrowdTrack and CityTrack, and $$\hat F_{\mathrm{view}_i}^{t}\in\mathbb{R}^{h\times w\times d}$$8 for Wildtrack and MultiviewX (Zhang et al., 21 Apr 2026). The explicit use of different matching thresholds reflects the distinction between large-scene real-world datasets and smaller established benchmarks.

6. Empirical results and ablations

On MVCrowdTrack, MVTrackTrans achieves MOTA 63.9%, IDF1 59.1%, MOTP 40.6%, MT 42.9%, and ML 8.2%, outperforming EarlyBird with MOTA 54.6%, MVFlow with 49.8%, and TrackTacular with 62.9% (Zhang et al., 21 Apr 2026). On CityTrack, MVTrackTrans achieves MOTA 55.4% and IDF1 34.4%, surpassing EarlyBird with MOTA 48.9%, MVFlow with 38.2%, and TrackTacular with 43.4% (Zhang et al., 21 Apr 2026).

On Wildtrack and MultiviewX, MVTrackTrans is reported as competitive with or exceeding several recent CNN and hybrid methods: it matches EarlyBird on MultiviewX, and rivals cutting-edge models such as MVTrajecter on Wildtrack when augmented with two-frame fusion and a simple Kalman filter (Zhang et al., 21 Apr 2026).

The ablation studies isolate several design choices. Adding a pure 2D view-heatmap branch without cross-interaction degrades performance on CityTrack, with MOTA dropping from 54.9 to 53.2. Introducing the View–Ground Interaction recovers and improves accuracy to MOTA 55.4 and IDF1 34.4. Supervising via dense BEV heatmaps far outperforms direct query-based coordinate regression, with MOTA 55.4 versus 40.7 (Zhang et al., 21 Apr 2026).

These ablations constrain the interpretation of the model’s gains. They indicate that the principal improvement is not attributable merely to adding 2D supervision or to using Transformers in a generic sense; rather, it is tied to the specific interaction between view queries and ground-plane track queries and to dense BEV heatmap supervision.

7. Significance, practical considerations, and limitations

MVTrackTrans demonstrates that a transformer-centric design, which tightly couples multi-view appearance cues with BEV temporal reasoning, scales more gracefully to large scenes with severe occlusion and prolonged occluded trajectories (Zhang et al., 21 Apr 2026). By explicitly sampling and cross-attending between ground queries and view queries, the model maintains identity-consistent feature embeddings even when a pedestrian is heavily occluded in some cameras. The learned uncertainty weighting further stabilizes training across the dual detection–tracking objectives.

The paper identifies several limitations and future directions: extension to fully end-to-end 3D trajectory estimation by lifting BEV predictions to full 3D paths; incorporation of learnable camera-parameter embeddings to handle dynamic or asynchronous camera rigs; and exploration of longer temporal windows greater than 2 frames in the cross-attention to strengthen associations across abrupt motion (Zhang et al., 21 Apr 2026).

A common misconception in this area is that multi-view crowd tracking improvements follow directly from stronger image backbones or denser per-view supervision. The reported ablations do not support that simplified view. Instead, the results suggest that architectural coupling between multi-view appearance cues and ground-plane temporal reasoning is the decisive factor in this design. More broadly, MVTrackTrans situates Transformer-based multi-view tracking within a regime where large scene scale, synchronization, calibration, and prolonged occlusion are first-order considerations rather than secondary benchmarking details.