DepTR-MOT: Depth-Aware Multi-Object Tracking

• The paper introduces DepTR-MOT, a DETR-based framework that integrates instance-level depth to refine association in challenging tracking scenarios.
• The method leverages a depth decoding branch and foundation-model supervision to address occlusions and close-proximity interactions without extra inference cost.
• Experimental results show improved HOTA and IDF1 scores in dense scenes, validating the approach's effectiveness for robust robotic perception.

DepTR-MOT is a depth-aware multi-object tracking framework introduced as a DETR-based detector enhanced with instance-level depth information and used within a tracking-by-detection pipeline to address occlusion, close-proximity interaction, and dense-scene failure modes that are common in robotic perception (Deng et al., 22 Sep 2025). Its central premise is that conventional 2D cues—bounding boxes, IoU, motion models, and appearance features—are often insufficient when targets overlap heavily in image space, whereas instance-level depth can disambiguate targets that are similar in 2D position, motion, or appearance. The framework is trained without ground-truth depth labels by using foundation models only during training, and it outputs per-instance depth at inference without requiring foundation models and without additional computational cost (Deng et al., 22 Sep 2025).

1. Problem Setting and Conceptual Framing

Visual multi-object tracking in the tracking-by-detection paradigm is typically decomposed into frame-wise detection and inter-frame association. In the setting targeted by DepTR-MOT, standard 2D cues become unreliable under occlusions, close-proximity interactions, and dense scenes, particularly in robotic environments such as sidewalks observed from quadruped robots with many pedestrians (Deng et al., 22 Sep 2025). The failure mode is straightforward: when two boxes overlap substantially, 2D IoU and positional similarity cannot reliably determine which target corresponds to which trajectory, and identity switches, fragmented trajectories, and association failures follow.

DepTR-MOT recasts this problem as “depth-informed trajectory refinement.” The method uses depth as a third dimension for disentangling targets that occupy similar 2D support but differ in distance from the camera. In the formulation described for the method, instance-level depth permits separation of overlapping boxes by depth, reasoning about ordering along the camera ray, and mitigation of false association when targets exhibit near-identical appearance or 2D motion (Deng et al., 22 Sep 2025).

A common misconception is to treat the method as a fully end-to-end video transformer tracker. The published system is instead a depth-aware DETR-based detector whose outputs are plugged into standard trackers such as SORT, OC-SORT, ByteTrack, and HybridSORT, with depth used to refine association cost in a second-stage matching step (Deng et al., 22 Sep 2025).

2. Detector Architecture and Depth Decoding

The detector is based on HGNetv2 as backbone and a D-FINE/Deformable DETR style encoder-decoder, with per-query outputs comprising class label, bounding box, and scalar depth (Deng et al., 22 Sep 2025). The architectural novelty lies in integrating a depth decoding branch on top of the localization decoder rather than attaching an external depth network. This branch uses predicted box centers as anchors and refines per-query depth through multi-head depth-aware attention over encoder features.

The depth branch introduces an additional head in the detection head that predicts scalar depth for each query. For query $q$, batch $b$, head $h$, and point $p$, the learned depth correction is

$\delta^d_{b,q,h,p} = \Phi(Q_{b,q,h,p}),$

and the depth-aware weight is

$$W_d = \sum_{p=1}^{P_l} \alpha_{b,q,h,p}\,\beta\,\big(o^d_{b,q,h,p} + \delta^d_{b,q,h,p}\big).$$

Feature sampling is performed at the normalized box center $\mathcal{T}(c_{b,q})$, and the head-wise outputs are concatenated before layer-wise refinement. The paper further describes depth refinement across decoder layers through transformer updates and residual depth prediction, with initialization $O^0_{b,q}=P^0_{b,q}=0$ (Deng et al., 22 Sep 2025).

A notable implementation choice is that box centers guide depth sampling while sampling offsets are disabled to avoid misalignment when boxes are inaccurate. This couples depth estimation tightly to localization while making the depth prediction branch robust and aligned with detection (Deng et al., 22 Sep 2025). The design goal is not generic monocular depth estimation, but target-aligned instance-level depth prediction that can be consumed directly by downstream association.

3. Foundation-Model Supervision Without Depth Annotations

DepTR-MOT is trained on datasets such as DanceTrack and QuadTrack, neither of which contains depth annotations. Supervision is therefore derived from two foundation models used only during training: Video Depth Anything for temporally consistent dense depth maps and SAM2 for prompt-based instance masks (Deng et al., 22 Sep 2025).

Given a batch of sequences $$\mathbf{X} \in \mathbb{R}^{B \times T \times 3 \times H \times W}$$, Video Depth Anything produces dense depth maps $D=\{D_{b,t}\}$. For each annotated 2D box $b$0, SAM2 provides a binary mask $b$1, from which an instance depth map $b$2 is formed. The instance-level soft depth label is then the mask-weighted average

$b$3

This removes background and occluding neighbors within box boundaries so that the soft label is focused on the target rather than the entire box support (Deng et al., 22 Sep 2025).

Training combines two depth-specific objectives with the standard DETR box/classification loss. The numerical depth alignment term is the depth regression loss $b$4, and the dense-teacher consistency term is the feature alignment loss $b$5, which uses cosine dissimilarity between projected detector encoder features and teacher depth features. The overall objective is

$b$6

The dense-depth distillation component is used to maintain global depth consistency. Instance labels alone are sparse, whereas dense depth maps encode scene-level near/far structure; the alignment objective therefore encourages the student encoder to respect relative scales and ordering across the full image (Deng et al., 22 Sep 2025). A plausible implication is that the method is optimized not only for per-instance scalar estimation but also for frame-level coherence, which is relevant when multiple targets must be compared during association.

The reported training setup uses HGNetv2, AdamW with learning rate $b$7, input resolution $b$8 for DanceTrack and $b$9 for QuadTrack, 5 epochs, batch size 1, and 2× RTX 3090. The depth teacher uses sliding windows, and ablation is reported as robust to window/stride variations (Deng et al., 22 Sep 2025).

4. Tracking-by-Detection and Depth-Informed Association

At inference, DepTR-MOT performs frame-wise detection with boxes and depth and then relies on standard trackers with Kalman filtering and Hungarian matching. The modification is to introduce depth distance into the association cost during second-stage matching (Deng et al., 22 Sep 2025).

Let tracks be $h$0, detections be $h$1, and scalar depths be $h$2 and $h$3. The depth distance matrix is defined as

$h$4

where $h$5 is a normalization factor derived from Video Depth Anything’s scale. If $h$6 denotes the original cost matrix computed from IoU, motion, and appearance, the refined cost becomes

$h$7

Hungarian matching is then applied to $h$8 (Deng et al., 22 Sep 2025).

This mechanism is designed for precisely those cases where 2D similarity is ambiguous. In crossing trajectories, two people can overlap in image space while remaining separable in depth. Under occlusion, the occluder and the occluded target differ in depth, so the additional cue can preserve identity when box motion becomes unreliable. In QuadTrack, panoramic distortion further weakens Euclidean 2D motion assumptions, and depth provides an alternative ordering cue (Deng et al., 22 Sep 2025).

The paper reports that moderate depth weighting, with $h$9, consistently improves performance, while excessively large $p$0 overweights depth and harms association when depth is noisy (Deng et al., 22 Sep 2025). This suggests that depth is treated as a complementary cue rather than as a replacement for appearance, IoU, or motion.

5. Empirical Results, Ablations, and Deployment Characteristics

Experiments are reported on QuadTrack and DanceTrack, with the headline HOTA scores of 27.59 and 44.47, respectively, for the depth-aware system (Deng et al., 22 Sep 2025). The QuadTrack results are emphasized because the dataset is a robotic platform MOT benchmark with omnidirectional imagery, pedestrians and vehicles, 10 FPS, and panoramic distortion.

Before summarizing the quantitative results, it is important to note the evaluation pattern described in the paper: the gains are especially large in association-oriented metrics such as IDF1 and AssA, which is consistent with the method’s stated role as trajectory refinement rather than a pure detector replacement (Deng et al., 22 Sep 2025).

Dataset	Setting	Reported result
QuadTrack	ByteTrack + DepTR with depth	HOTA 27.59, IDF1 28.035, MOTA -3.6548, AssA 26.553
DanceTrack	ByteTrack + DepTR with depth	HOTA 44.465, IDF1 47.418, AssA 28.618
QuadTrack	Average across trackers	+2.2 HOTA, +2.9 IDF1 vs 2D-only setups

On QuadTrack, ByteTrack + DepTR with depth is reported at HOTA 27.59 versus 23.903 with D-FINE, IDF1 28.035 versus 23.072, MOTA $p$1 versus $p$2, and AssA 26.553 versus 20.364 (Deng et al., 22 Sep 2025). On DanceTrack, ByteTrack + DepTR with depth reaches HOTA 44.465 versus 42.343 with D-FINE, IDF1 47.418 versus 45.922, and AssA 28.618 versus 26.64 (Deng et al., 22 Sep 2025). The authors also report that overall FPS remains around 25–30 and that the depth branch does not add inference computation beyond the detector.

The ablation studies attribute the best behavior to the combination of the depth-aware weight $p$3, depth regression loss, and feature alignment loss. In the DanceTrack ablation, the full model achieves HOTA 44.939 and AssA 47.956, outperforming variants that include only part of the design (Deng et al., 22 Sep 2025). The reported fusion hyperparameter study identifies the best region around $p$4, $p$5, while $p$6 degrades HOTA and can produce large negative MOTA because depth noise begins to dominate the cost (Deng et al., 22 Sep 2025).

The deployment claims are correspondingly narrow and explicit: the framework uses Video Depth Anything and SAM2 during training, but at inference the model is a single network consisting of backbone, DETR transformer, detection head, and depth head, with complexity similar to D-FINE / Deformable DETR and reported real-time operation around 25–30 FPS (Deng et al., 22 Sep 2025).

6. Relation to Transformer-Based MOT, Limitations, and Prospective Extensions

DepTR-MOT occupies a distinct position within transformer-based MOT. TR-MOT, for example, decouples detection and association through a deformable transformer Reference Search module that uses previous track states as references and predicts one-to-one track state propagation across adjacent frames (Chen et al., 2022). MATR instead targets end-to-end DETR-style tracking and argues that joint processing of detection and track queries creates query collisions; its Motion-Aware Transformer updates track queries with explicit motion supervision before the main decoder (Yang et al., 26 Sep 2025). ContrasTR, by contrast, keeps the DETR detector architecture essentially intact and turns it into a tracker by learning identity-preserving embeddings with an instance-level contrastive loss and a lightweight assignment method (Plaen et al., 2023).

Against this background, DepTR-MOT is neither a reference-search association transformer in the TR-MOT sense, nor an end-to-end query-propagation tracker in the MATR sense, nor a pure appearance-embedding extension in the ContrasTR sense. Its specific contribution is to inject instance-level depth into a DETR-based detector and then use that depth inside otherwise standard tracking-by-detection systems (Deng et al., 22 Sep 2025). A common point of confusion is therefore terminological: “DETR-based MOT” is a broader category, while DepTR-MOT denotes a depth-informed trajectory refinement framework with pseudo-depth supervision.

The paper identifies three principal limitations. First, the method depends on teacher quality: failures or domain shift in Video Depth Anything can propagate errors into the learned depth predictor. Second, the depth is monocular and relative rather than metric, which can limit downstream tasks that require absolute 3D scale. Third, the framework remains a 2D tracking-by-detection pipeline rather than a fully end-to-end joint detection-tracking architecture (Deng et al., 22 Sep 2025).

The future directions discussed are correspondingly specific: extending the training strategy to end-to-end MOT frameworks, moving toward 3D MOT by combining instance depth with camera calibration, combining depth with pose estimation for richer human scene understanding, incorporating real RGB-D sensors during training or inference, and replacing the current teachers with more advanced foundation models such as DepthCrafter or Depth Anything V2 (Deng et al., 22 Sep 2025). This suggests a broader research program in which depth supervision is treated as an intermediate supervisory signal for association robustness even when explicit depth labels are absent.