CoopTrack: Cooperative 3D Tracking

• CoopTrack is a cooperative 3D multi-object tracking framework that fuses vehicle and infrastructure data through end-to-end instance-level association.
• It employs a fusion-after-decoding pipeline to separate semantic and motion features and integrates temporal cues for stable tracking across frames.
• The framework achieves improved mAP and AMOTA with efficient communication while addressing challenges like pose errors, latency, and duplicate detections.

CoopTrack is a cooperative 3D multi-object tracking framework for vehicle-infrastructure and, more generally, multi-agent sequential perception that uses an ego vehicle’s image stream together with compact, instance-level information transmitted from another agent such as roadside infrastructure. It is formulated as a fully instance-level end-to-end framework with learnable instance association, and is explicitly positioned against both single-frame cooperative perception and decoupled tracking-by-cooperative-detection pipelines. In the concrete setting studied, the task is vehicle-infrastructure cooperative 3D MOT from images, with outputs given as temporally consistent 3D boxes $\mathcal{B}_t = \{ b_t^i \}$, where $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$ (Zhong et al., 25 Jul 2025).

1. Problem formulation and design rationale

CoopTrack addresses cooperative sequential perception rather than single-frame cooperative detection. The distinction is central: in cooperative tracking, the system must solve multi-agent fusion and temporal data association simultaneously, preserve object identity across frames, avoid duplicate detections when both agents observe the same object, and avoid incorrect merges when different objects are mistakenly associated across agents. The paper further situates the problem under noisy inter-agent relative poses, different viewpoints and sensing domains, communication limitations, and temporal misalignment (Zhong et al., 25 Jul 2025).

The framework takes synchronized time-series sensor data from multiple agents and their relative poses over time. In the reported experiments, the modality is image-only vehicle-infrastructure cooperative 3D MOT. At each time step, the model returns object states with maintained IDs, including 3D center, size, yaw, planar velocity, and semantic class. This makes the tracking problem both geometric and temporal: object state estimation must remain stable under cross-agent appearance mismatch and under the partial observability characteristic of vehicle and roadside viewpoints.

A defining architectural choice is the “fusion-after-decoding” pipeline. Each agent first runs an end-to-end query-based tracker independently to decode local per-instance representations. Only after this local decoding are sparse instance-level features transmitted, aligned, associated, aggregated, and re-decoded cooperatively. This differs from prior “fusion-before-decoding” approaches, which mix features or queries earlier and can create ambiguity or conflict. The paper explicitly contrasts CoopTrack with tracking-by-cooperative-detection pipelines and with prior end-to-end cooperative tracking efforts such as UniV2X, which were limited by rule-based query association and fusion-before-decoding design (Zhong et al., 25 Jul 2025).

A plausible implication is that CoopTrack treats cooperation as an object-centric association problem rather than a dense feature-registration problem. The paper states this explicitly in communication terms: what is transmitted is sparse instance-level information rather than dense BEV maps or image feature tensors.

2. End-to-end architecture

The overall architecture has two subsystems, one on the vehicle side and one on the infrastructure side. At time $t$, the ego vehicle constructs its query set by concatenating propagated tracking queries from time $t-1$ with a fixed number of randomly initialized queries. Each query is

$$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$

where $f_t^i$ is the query feature and $p_t^i$ is a 3D reference point (Zhong et al., 25 Jul 2025).

On each side, the local tracker produces motion and semantic instance features through the Multi-Dimensional Feature Extraction module: $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$ with an analogous process on the infrastructure side yielding $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$. These infrastructure-side features are the transmitted objects of communication. The transmitted representation is therefore sparse and instance-centric rather than scene-dense (Zhong et al., 25 Jul 2025).

After communication, a Cross-Agent Alignment module transforms both agents’ feature spaces: $\widetilde{\mathcal{M}_t^{\mathrm V}, \widetilde{\mathcal{S}_t^{\mathrm V}, \widetilde{\mathcal{M}_t^{\mathrm I}, \widetilde{\mathcal{S}_t^{\mathrm I} = \mathbf{CAA} ( \mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V}, \mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}, \mathbf{R}, \mathbf{t} ).$ The aligned features are then passed to Graph-Based Association: $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$0 where $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$1 is an affinity matrix whose entry $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$2 indicates how likely vehicle instance $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$3 and infrastructure instance $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$4 correspond to the same object. Aggregation then produces fused motion and semantic features,

$b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$5

which are decoded into object states and categories (Zhong et al., 25 Jul 2025).

The tracking state is propagated temporally by reusing semantic features as the next track-query features and updating reference points using a constant-velocity assumption. This yields an integrated recurrent tracking mechanism rather than a post-hoc tracker attached to cooperative detections (Zhong et al., 25 Jul 2025).

3. Multi-Dimensional Feature Extraction

The Multi-Dimensional Feature Extraction module is intended to overcome two limitations attributed to standard query-based trackers: dependence on single-frame information and entanglement of semantics and motion within a single query representation. CoopTrack instead explicitly decouples semantic and motion representations and then enriches them temporally (Zhong et al., 25 Jul 2025).

The process begins with a query-based detector/tracker decoder that produces coarse 3D boxes $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$6 and updated query features $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$7. The implementation uses BEVFormer as the detector backbone, but the paper states that MDFE is agnostic to the particular query-based detector. Semantic features are extracted from updated query features through an MLP: $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$8 These semantic features encode appearance and class-related information at the instance level (Zhong et al., 25 Jul 2025).

Motion features are derived not from latent query embeddings but from the predicted coarse 3D boxes. For each predicted box $b_t^i = [x, y, z, w, l, h, \theta, v_x, v_y]$9, 3D corners

$t$0

are computed, converted to coordinates relative to the box center, flattened, and passed through a PointNet-style encoder consisting of a 4-layer MLP plus max pooling. The result is

$t$1

which is intended to capture geometric shape and 3D layout cues useful for box regression, motion reasoning, and cross-agent matching (Zhong et al., 25 Jul 2025).

Temporal information is injected into both semantic and motion features using a temporal transformer block with two decoder layers, inspired by PF-Track. Temporal sinusoidal positional encodings are added; shorter histories are zero-padded and masked; and historical features are stored in a sliding-window FIFO queue. The paper reports that increasing the number of historical frames from 0 to 4 improves performance substantially and that the benefit saturates beyond 4. The chosen history length is therefore 4, for which the ablation reports $t$2 mAP and $t$3 AMOTA; at 5 frames, mAP rises slightly to $t$4 but AMOTA drops to $t$5 (Zhong et al., 25 Jul 2025).

This design suggests that CoopTrack’s instance representation is intentionally factorized into semantically oriented and kinematically oriented subspaces, with temporal refinement applied to both rather than only to a single entangled track query.

4. Cross-agent alignment, association, and aggregation

Cross-Agent Association and Aggregation is the second major block. Its first step, Cross-Agent Alignment, addresses the domain gap induced by different viewing angles, positions, and sensing conditions between vehicle and infrastructure. For motion features, the latent alignment is written as

$t$6

while explicit reference-point transformation is

$t$7

Here $t$8 and $t$9 are learnable latent-space rotation and translation, whereas $t-1$0 and $t-1$1 are the physical relative pose. The semantic features follow the same latent transformation pipeline. The physical rotation matrix $t-1$2 is converted into a 6D continuous rotation representation and, together with $t-1$3, is fed into two MLPs to predict the latent transform parameters. The paper adopts segmented mapping from STAR-Track to reduce parameters and ease training (Zhong et al., 25 Jul 2025).

Graph-Based Association is the framework’s learnable instance association mechanism. A fully connected bipartite graph

$t-1$4

is constructed between vehicle instances and infrastructure instances. Vehicle node features are formed by concatenating motion and semantic features and passing them through an MLP to obtain

$t-1$5

with infrastructure nodes defined analogously. Edge features are derived from pairwise reference-point differences: $t-1$6 An MLP maps these spatial differences to edge features $t-1$7 (Zhong et al., 25 Jul 2025).

Association scores are then computed by graph attention: $t-1$8 followed by a feed-forward network and sigmoid: $t-1$9 For actual pairing, the framework still uses the Hungarian algorithm with cost

$$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$0

The important distinction is that the affinities are learned end-to-end from semantics, geometry, temporal cues, and relative position, rather than being set by hand-crafted Euclidean thresholds or fixed positional rules (Zhong et al., 25 Jul 2025).

The aggregation stage adaptively fuses matched cross-agent instances and retains unmatched unique instances to widen scene coverage. The paper does not provide a closed-form equation for the internal fusion operator of $$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$1, but describes it conceptually as merging aligned representations for matched objects while preserving unmatched objects. A plausible implication is that duplicate suppression and coverage expansion are treated jointly rather than as separate downstream operations.

5. Training procedure, implementation, and datasets

Training is two-stage. In stage 1, the vehicle-side and infrastructure-side end-to-end tracking models are trained separately with their own ground truth. Classification uses Focal Loss with $$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$2, $$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$3, box regression uses L1 loss, and the objective is

$$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$4

with

$$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$5

In stage 2, the cooperative model is initialized from stage-1 pretrained models and trained with cooperative ground truth plus generated association labels. Association is supervised as binary classification over the affinity matrix using Focal Loss with $$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$6, $$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$7, under

$$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$8

where

$$q_t^i = \{ f_t^i \in \mathbb{R}^{1 \times d},\; p_t^i \in \mathbb{R}^{1 \times 3} \},$$9

Association labels are generated automatically: if the same ground-truth object is matched by both a vehicle-side prediction and an infrastructure-side prediction, the corresponding entry in the association label matrix is positive; otherwise it is negative (Zhong et al., 25 Jul 2025).

The detector/tracker backbone is BEVFormer with image inputs only. Main experiments use ResNet50 or ResNet101, with an additional ConvNeXt-Small result in the appendix. Previous-frame active instances are propagated as track queries, semantic features become query embeddings for the next frame, and positions are advanced by constant-velocity motion. The CAA module uses MLPs to map physical pose to latent transform parameters, and GBA functions as a graph-attention association head over the fully connected cross-agent instance graph (Zhong et al., 25 Jul 2025).

On V2X-Seq, the vehicle perception range is $f_t^i$0 meters horizontally and $f_t^i$1 in height, while the roadside range is $f_t^i$2 horizontally with the same height range. On Griffin, both vehicle and UAV use $f_t^i$3 horizontally, while UAV height range is $f_t^i$4. For ResNet50, images are cropped to $f_t^i$5 and BEV feature size is $f_t^i$6; for ResNet101, the original image size is kept and the BEV size is $f_t^i$7. Optimization uses AdamW with weight decay $f_t^i$8, initial learning rate $f_t^i$9, and cosine annealing. To reduce memory, the paper uses streaming video training. On V2X-Seq, first-stage training is 48 epochs for vehicle and 24 epochs for infrastructure, followed by 48 epochs for stage 2; Griffin uses 48 + 48 epochs. Experiments run on NVIDIA 3090 GPUs (Zhong et al., 25 Jul 2025).

The primary evaluation datasets are V2X-Seq and Griffin. V2X-Seq is described as a real-world vehicle-infrastructure cooperative sequential perception dataset with around 100 dynamic scenes, each 10–25 seconds at 10Hz; evaluation is performed on both native 10Hz data and a 2Hz downsampled version. Griffin is an aerial-ground cooperative dataset in Carla, evaluated on Griffin-25m with the official splits. Primary metrics are mAP and AMOTA, with ATE, ASE, AOE, AVE, AMOTP, IDS, and communication cost in bytes per second reported in some experiments (Zhong et al., 25 Jul 2025).

6. Empirical performance, limitations, and research context

The headline result on V2X-Seq 2Hz is $p_t^i$0 mAP and $p_t^i$1 AMOTA, reported in table form as

$p_t^i$2

with transmission cost

$p_t^i$3

Compared with UniV2X at $p_t^i$4 mAP and $p_t^i$5 AMOTA with similar communication cost $p_t^i$6, CoopTrack improves by $p_t^i$7 mAP and $p_t^i$8 AMOTA. On Griffin, it reports $p_t^i$9 mAP and $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$0 AMOTA with cost $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$1, outperforming UniV2X’s $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$2 mAP and $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$3 AMOTA while remaining far below early fusion’s communication cost $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$4 (Zhong et al., 25 Jul 2025).

On V2X-Seq 10Hz, CoopTrack with ResNet50 reaches

$$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$5

while with ResNet101 it obtains

$$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$6

Against UniV2X-Track, the paper reports higher mAP and AMOTA together with substantially fewer ID switches (Zhong et al., 25 Jul 2025).

Ablation results attribute gains to all major components. Starting from a $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$7 mAP, $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$8 AMOTA baseline, adding the proposed pipeline gives $$\mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V} = \mathbf{MDFE} \big( F_t^{\mathrm V}, \mathcal{Q}_t^{\mathrm V}, \mathcal{M}_{t-\tau:t-1}^{\mathrm V}, \mathcal{S}_{t-\tau:t-1}^{\mathrm V} \big),$$9; adding MDFE yields $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$0; adding CAA yields $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$1; adding GBA yields $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$2; combining pipeline + MDFE + CAA reaches $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$3; and the full model reaches $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$4. Among single modules, CAA appears especially important, while GBA contributes strongly in combination, particularly for AMOTA (Zhong et al., 25 Jul 2025).

Communication efficiency is one of the framework’s explicit claims. Dense BEV fusion methods are described as requiring on the order of $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$5 to $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$6 bytes per second, whereas CoopTrack operates in the $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$7 to $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$8 range depending on setting. The appendix runtime analysis reports that with ResNet50, MDFE takes 42.01 ms, CAA 2.08 ms, and GBA+aggregation 9.95 ms, for a total of 121.88 ms; with ResNet101, the total is 207.99 ms. The authors state that the ResNet50 model is nearly real-time at about 10Hz (Zhong et al., 25 Jul 2025).

The paper also identifies clear limitations. Training is relatively complex because of the two-stage design and generated association labels. The framework relies on reasonably accurate local instance proposals; if local object extraction fails, sparse instance communication cannot recover what was never represented. The method is sensitive to communication latency and to global pose errors through reference-point misalignment. In robustness experiments, a 500 ms delay causes about 15.3% mAP and 31.8% AMOTA degradation, although a feature flow prediction module from prior work greatly reduces this degradation. The aggregation operator is described more conceptually than analytically, and the main setting uses one-way communication from infrastructure to vehicle (Zhong et al., 25 Jul 2025).

Within the broader literature, CoopTrack occupies a distinct position. DMSTrack formulates cooperative tracking as differentiable multi-sensor Kalman filtering with per-detection uncertainty estimation in a late-fusion setting (Chiu et al., 2023). SparseCoop likewise targets cooperative 3D detection and tracking, but replaces dense intermediate representations with kinematic-grounded sparse queries and reports $$\mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}$$9 AMOTA on V2X-Seq versus CoopTrack’s $\widetilde{\mathcal{M}_t^{\mathrm V}, \widetilde{\mathcal{S}_t^{\mathrm V}, \widetilde{\mathcal{M}_t^{\mathrm I}, \widetilde{\mathcal{S}_t^{\mathrm I} = \mathbf{CAA} ( \mathcal{M}_t^{\mathrm V}, \mathcal{S}_t^{\mathrm V}, \mathcal{M}_t^{\mathrm I}, \mathcal{S}_t^{\mathrm I}, \mathbf{R}, \mathbf{t} ).$0 under lower transmission cost (Wang et al., 7 Dec 2025). These comparisons suggest that CoopTrack’s principal historical role is not merely its benchmark numbers, but its explicit reformulation of cooperative tracking as end-to-end instance association and aggregation after local query decoding, with sparse instance-level communication as the system primitive (Zhong et al., 25 Jul 2025).