MVCrowdTrack: Dual-Mode Multi-View Tracking

• MVCrowdTrack is a dual-usage framework that combines a large-scale outdoor multi-camera benchmark for pedestrian tracking with a crowdsourced markerless motion capture system.
• The benchmark offers 4,122 labeled frames from 7 calibrated cameras over an 18-minute campus scene, delivering detailed ground-plane trajectories of 342 pedestrians.
• The associated MVTrackTrans model leverages Transformer-based fusion of multi-view features to maintain tracking identities through heavy occlusion.

Searching arXiv for MVCrowdTrack and directly related papers. MVCrowdTrack appears in arXiv literature in two distinct technical senses. In its recent and more prominent usage, it is a newly collected, large-scale real-world multi-view crowd-tracking benchmark introduced together with the Transformer-based MVTrackTrans model for estimating each person’s tracking trajectories on the ground of the scene (Zhang et al., 21 Apr 2026). In an earlier usage, MVCrowdTrack denotes a crowdsourced markerless motion-capture pipeline in which information workers repeatedly click body joints in video frames to produce 2D structure over time and, with multiple views, 3D reconstructions (Spiro et al., 2012). The two usages are related by their dependence on human motion estimation from video, but they target different problem formulations: ground-plane multi-object tracking in large outdoor scenes versus markerless body-joint tracking and reconstruction.

1. Scope and dual usage

The term MVCrowdTrack has been used for both a dataset-centered multi-view crowd-tracking benchmark and a human-in-the-loop motion-capture system.

Usage	Primary object	Principal output
MVCrowdTrack in 2026	Large-scale real-world multi-view crowd-tracking benchmark	Ground-plane pedestrian trajectories
MVCrowdTrack in 2012	Crowdsourced markerless motion-capture pipeline	2D joint trajectories and optional 3D structure

In the 2026 formulation, the benchmark is explicitly designed for large real-world scenes in which scene size and occlusion are more complicated than in small-scene datasets such as Wildtrack and MultiviewX (Zhang et al., 21 Apr 2026). The associated model, MVTrackTrans, operates on the ground plane and uses interactions between camera views and the ground plane. In the 2012 formulation, MVCrowdTrack ingests raw video sequences, uses a web-based annotation interface deployed via Amazon Mechanical Turk, and aggregates redundant annotations by a median-based consensus to produce searchable and visualizable 2D trajectories (Spiro et al., 2012).

This dual usage indicates that MVCrowdTrack is not a single immutable framework. Rather, the name spans two research regimes: one centered on human annotation pipelines for articulated motion, and one centered on benchmark construction and Transformer-based multi-view crowd tracking. A plausible implication is that the later usage broadens the term from annotation software to a standardized evaluation substrate for large-scene multi-camera tracking.

2. Dataset specification and geometric representation

MVCrowdTrack, as introduced with MVTrackTrans, is a newly collected, large-scale real-world multi-view crowd-tracking benchmark consisting of 1 campus scene, specifically an outdoor pedestrian area of size $120\,\mathrm m \times 80\,\mathrm m$, recorded by 7 synchronized, calibrated RGB cameras with image size $5312 \times 2988$ (Zhang et al., 21 Apr 2026). The raw frame rate is 60 fps, while labeling is performed at 4 fps. The benchmark contains 4,122 labeled multi-view frames, corresponding to approximately 18 minutes at 4 fps, with 342 unique pedestrian IDs and an average track length of 176 frames. The provided split is 80% train, or 3,297 frames, and 20% test, or 825 frames.

The dataset is grounded in an explicit camera and world-coordinate model. Intrinsics and extrinsics are pre-computed for each camera as $K[R\mid T]$. The ground plane is discretized into a 2D map of size $1200 \times 800$ pixels, with each pixel representing 0.1 m in the real world. For a 3D ground point $(x,y,z)$ with $z=0$ at ground, the corresponding image coordinates $(u,v)$ satisfy

$\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$

The data collection environment is a campus plaza with crowd density up to a few hundred people over the entire 18 min and moderate to high local occlusion. The stated occlusion patterns are naturally occurring: pedestrians overlap, occlude each other behind street furniture and trees, and exhibit frequent view-to-view occlusions. These conditions make the benchmark specifically relevant for multi-view tracking methods that must sustain identity continuity in large outdoor scenes.

3. Annotation protocol, data format, and recommended usage

The annotation pipeline is fully manual at the view level. Bounding boxes in each camera view are labeled using VGG-VIA, and consistent ID assignment is maintained across time and across cameras (Zhang et al., 21 Apr 2026). For ground-truth localization, the foot-point of each bounding box is projected via each camera’s calibration, and the average of all valid per-view projections is taken as the ground-truth 2D location. Multiple annotators cross-check ID consistency, and any ambiguous frames are relabeled until agreement.

The suggested dataset structure includes per-camera image directories, calibration files, annotation JSON files, and ground-plane heatmaps. The recommended split is the provided 80%/20% train/test division. Each JSON entry contains a frame index; for each camera, a list of objects of the form $$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$; and a ground-truth list of $\{x,y,\texttt{id}\}$ in ground-map coordinates (Zhang et al., 21 Apr 2026).

The practical loading and preprocessing procedure is specified explicitly. One reads calibration files $5312 \times 2988$0 for each camera; for each labeled frame $5312 \times 2988$1, loads images from cam0 through cam6; loads bounding boxes and IDs from the annotation JSON; converts each bounding-box foot centre to ground $5312 \times 2988$2 via inversion of $5312 \times 2988$3; constructs a Gaussian heatmap $5312 \times 2988$4; and resizes images to $5312 \times 2988$5 with normalization for ResNet (Zhang et al., 21 Apr 2026). During training, two consecutive frames’ multi-view images are fed into the MVTrackTrans pipeline.

These design choices place the benchmark at the intersection of calibrated multi-view geometry, dense crowd observation, and ground-plane representation. The explicit ground-map discretization and synchronized multi-camera setup also make the dataset suitable for methods that reason jointly over image evidence and world-space occupancy.

4. MVTrackTrans architecture

MVTrackTrans is a Transformer-based multi-view tracking framework operating on the ground plane and organized into three stages: Feature Extraction & Fusion, Tracking Encoding, and Tracking Decoding (Zhang et al., 21 Apr 2026). The backbone is ResNet-18 with FPN, which produces single-view feature maps $5312 \times 2988$6 for cameras $5312 \times 2988$7.

Feature extraction and multi-view fusion are performed through multi-height sampling. A set of 3D voxels $5312 \times 2988$8 is defined over the ground extent and several heights $5312 \times 2988$9, and each voxel is projected into each camera according to

$K[R\mid T]$0

Bilinear sampling of $K[R\mid T]$1 is then carried out at each projected corner, and aggregation across 8 vertices and all views yields a voxel feature. After collapsing the height axis and applying convolutional fusion, the model obtains multi-scale ground features $K[R\mid T]$2.

The encoder uses Multi-scale Deformable Attention from Deformable DETR to process $K[R\mid T]$3 and $K[R\mid T]$4 independently, producing query sets $K[R\mid T]$5 and $K[R\mid T]$6. Track query sampling extracts $K[R\mid T]$7 from $K[R\mid T]$8 at previous detection locations $K[R\mid T]$9. In parallel, view queries are sampled from each view’s detection backbone, yielding $1200 \times 800$0, which are concatenated into $1200 \times 800$1. View-Ground Interaction is implemented by cross-attention: $1200 \times 800$2 with per-head attention

$1200 \times 800$3

The decoder has two parallel branches. The offset decoder applies MSDA to refined track queries and current-frame ground queries: $1200 \times 800$4 followed by an MLP head

$1200 \times 800$5

The heatmap decoder upsamples $1200 \times 800$6 via FPN and applies a convolutional head to generate the ground heatmap $1200 \times 800$7. Training uses focal loss for the ground heatmap and image-view heatmap, an $1200 \times 800$8 offset regression loss on valid centres, and an uncertainty-weighted total loss: $1200 \times 800$9

$(x,y,z)$0

$(x,y,z)$1

The architectural emphasis is therefore on explicit view-ground interactions rather than isolated per-camera processing. This suggests that the model treats ground-plane tracking as a coupled inference problem over image evidence, calibration geometry, and temporal identity propagation.

5. Evaluation protocol and reported performance

Evaluation on MVCrowdTrack is conducted on the ground plane with association radius $(x,y,z)$2 (Zhang et al., 21 Apr 2026). The reported metrics are MOTA, defined as

$(x,y,z)$3

MOTP as the average localization precision of matched tracks, IDF1,

$(x,y,z)$4

and the trajectory coverage measures MT, the percentage of tracks matched for more than 80% of their lifespan, and ML, the percentage matched for less than 20%.

On MVCrowdTrack, EarlyBird reports MOTA $(x,y,z)$5, MOTP $(x,y,z)$6, IDF1 $(x,y,z)$7, MT $(x,y,z)$8, and ML $(x,y,z)$9. MVFlow reports MOTA $z=0$0, MOTP $z=0$1, IDF1 $z=0$2, MT $z=0$3, and ML $z=0$4. TrackTacular reports MOTA $z=0$5, MOTP $z=0$6, IDF1 $z=0$7, MT $z=0$8, and ML $z=0$9. MVTrackTrans reports MOTA $(u,v)$0, MOTP $(u,v)$1, IDF1 $(u,v)$2, MT $(u,v)$3, and ML $(u,v)$4 (Zhang et al., 21 Apr 2026). On the companion CityTrack benchmark, the same paper reports MOTA $(u,v)$5, MOTP $(u,v)$6, IDF1 $(u,v)$7, MT $(u,v)$8, and ML $(u,v)$9 for MVTrackTrans.

The qualitative analysis states that long-term ground trajectories show that MVTrackTrans maintains identity continuity through heavy occlusions and tracks more individuals than EarlyBird or TrackTacular, especially over extended time spans (Zhang et al., 21 Apr 2026). The paper interprets the aggregate results as evidence that the Transformer-based design is better suited than CNN-based baselines for large, occlusion-rich scenes. At minimum, the reported numbers establish that MVCrowdTrack is not merely a data collection effort; it is paired with a concrete performance baseline and an evaluation regime defined in ground-plane terms.

6. Earlier MVCrowdTrack: crowdsourced markerless motion capture and relation to crowd video analysis

In the earlier work "Markerless Motion Capture in the Crowd" (Spiro et al., 2012), MVCrowdTrack is a web-based crowdsourcing system for obtaining motion capture data from video recordings. It ingests raw video sequences such as $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$0 clips of 100–200 frames, and workers use a JavaScript/HTML5 annotation interface via Amazon Mechanical Turk to place keyframes for specified body joints including HEAD, LANK, and RELB. A single worker tracks 3–4 points over approximately 100 frames in 5–10 minutes. The output for each camera view is a set of 2D trajectories $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$1, which are stored, indexed for search, visualized as stick figures or motion traces, or used as input to 3D reconstruction.

The annotation protocol uses a keyframe paradigm in which workers specify sparse $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$2 tuples and linear or Bézier interpolation fills intermediate frames. The interpolation rule for joint $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$3 between keyframes $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$4 and $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$5 is

$\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$6

applied separately to $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$7 and $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$8. To improve throughput and reduce cursor motion, all common actions have keyboard shortcuts, explicitly motivated by Fitts’ Law. Part labeling uses 4-letter codes with textual descriptions and colored squares, and examples such as “Hand wearing glove” versus “throwing hand” are used to prevent left–right confusion. Worker qualification requires a score of at least 80% on an auto-scored 90-frame tracking task compared against an expert key.

Redundant assignments are central to the method. Each video or HIT is typically assigned to 5–30 workers, and for each joint $\begin{pmatrix}u\v\1\end{pmatrix} = K\,[R\mid T]\, \begin{pmatrix}x\y\z=0\1\end{pmatrix}.$9 and frame $$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$0, the consensus estimate is the median across workers,

$$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$1

which discards extreme outliers automatically (Spiro et al., 2012). Optional post-aggregation smoothing can be imposed through a low-pass filter or cubic-spline regularization,

$$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$2

although the paper notes that simple linear interpolation plus manual Bézier edits often suffices. The 2D aggregation can also be viewed through the implicit loss

$$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$3

with the median estimator approximating an $$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$4-robust solution.

For multiple synchronized camera views, 3D reconstruction is formulated through a projection model

$$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$5

and a bundle-adjustment style objective

$$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$6

where $$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$7 enforces temporal smoothness or low-rank nonrigid priors, for example as in Torresani et al. 2008 (Spiro et al., 2012). The paper also describes a factorization-based Tomasi–Kanade-style approach based on the measurement matrix $$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$8 and SVD, followed by metric constraints to recover $$\{ \texttt{bbox}:[x_1,y_1,x_2,y_2], \texttt{id}:\texttt{int} \}$$9, $\{x,y,\texttt{id}\}$0, and $\{x,y,\texttt{id}\}$1.

The reported results for a 13-point baseball pitch annotation over 100 frames give average pixel error of approximately 8–15 px, with 5 px corresponding to approximately 2.5 cm, and motion error of 1.2–4.3 px/frame, with slower joints yielding lower error (Spiro et al., 2012). A crowd-size study reports that error drops rapidly up to $\{x,y,\texttt{id}\}$2 workers, with further gains of approximately 0.01 px per additional worker. Sample outputs include static motion summaries, stick figure animations, a gesture-based video search index using head and hand motion hashes, and 3D animated reconstructions of MLB pitchers after artist cleanup. The stated limitations are equally explicit: accuracy is limited by video quality and worker precision; major errors occur when low contrast occludes joints; costs increase beyond 5 workers; the system lacks integrated auto-tracking; and uncalibrated or single-view 3D remains challenging.

A related but distinct line of work on crowd video analysis formulates simultaneous tracking, group detection, and multi-level activity recognition in a unified causal pipeline under a linear programming formulation for hidden-structure estimation and a structured SVM for collective, group, and atomic activity labels (Bhargava et al., 2017). This suggests a broader progression across the literature: from crowd-powered annotation and reconstruction, to online structured inference over tracks and groups, and then to large-scale calibrated multi-view benchmarks paired with Transformer-based ground-plane tracking.