Depth-Temporal Trajectory Modeling

• Depth-Temporal Trajectory Modeling is a framework where depth is treated as a dynamic state evolving over time, enabling multi-frame depth forecasting and tracking.
• It unifies various approaches such as RGB video forecasting, event-guided motion estimation, and depth integration in multi-object tracking pipelines.
• Incorporating spatiotemporal attention and dynamical state representations, DTM improves stability and accuracy in depth estimation across dynamic scenes.

Searching arXiv for recent and directly relevant papers on “Depth-Temporal Trajectory Modeling” and related formulations. Depth-Temporal Trajectory Modeling (DTM) denotes a family of formulations in which depth is modeled as a temporally evolving quantity rather than as an isolated frame-wise estimate. In the literature considered here, the term spans several distinct settings: future depth-sequence forecasting from RGB video, event-guided parametric trajectory fitting for 3D motion estimation, instance-level depth refinement for multi-object tracking, Kalman-filtered depth states for panoramic tracking, and joint recovery of metric depth with dense camera-rotation trajectories from perspective-based blur (Boulahbal et al., 2023, Wan et al., 14 Mar 2025, Deng et al., 22 Sep 2025, Deng et al., 29 Jun 2026, Qiu et al., 9 Dec 2025). A related trajectory-generation line factorizes global and local latent variables under spatiotemporal-validity constraints, which does not itself model depth but is relevant to the temporal-structure side of the problem (Zhang et al., 2020).

1. Taxonomic scope

The surveyed work does not present DTM as a single standardized algorithm. Instead, the common thread is that depth participates in temporal structure: as a predicted sequence, as a latent dynamical state, as a trajectory-conditioned geometric cue, or as a quantity coupled to image formation over time. This suggests that DTM is best understood as a modeling perspective rather than a fixed architecture.

Work	Observation domain	DTM object
STDepthFormer	RGB video	forecast depth maps $D_t, D_{t+1}, D_{t+3}, D_{t+5}$
EMoTive	event stream	event-guided non-uniform rational curves for motion and depth motion
DepTR-MOT	MOT detection/tracking	instance-level depth integrated into matching and state update
CylindTrack	panoramic MOT	Kalman-filtered depth state $[d_t^i,\dot d_t^i]^\top$
Perspective-based blur method	blurred monocular video	dense rotation trajectory $\Theta$ coupled to metric depth map $L$

In STDepthFormer, four consecutive RGB images $I_{t-3}, I_{t-2}, I_{t-1}, I_t$ resized to $192\times 640$ are mapped to four forecast depth outputs $D_t, D_{t+1}, D_{t+3}, D_{t+5}$, with each depth produced at four pyramid levels (Boulahbal et al., 2023). In EMoTive, DTM refers to modeling spatio-temporal trajectories via event-guided non-uniform parametric curves, followed by multi-temporal sampling to recover optical flow and depth motion fields (Wan et al., 14 Mar 2025). In DepTR-MOT and CylindTrack, DTM operates inside tracking-by-detection pipelines: the former predicts per-instance depth and fuses it into association and 3D state updates, whereas the latter promotes depth to a trajectory-level state filtered by a one-dimensional constant-velocity Kalman filter (Deng et al., 22 Sep 2025, Deng et al., 29 Jun 2026). In the perspective-blur setting, DTM denotes a two-stage pipeline that estimates metric depth and densifies a sparse optical trajectory into a dense rotation field (Qiu et al., 9 Dec 2025).

2. State representations and mathematical formulations

A central distinction across DTM formulations is the choice of state space. STDepthFormer uses depth maps as the temporal state. Its network predicts a disparity-like activation $\sigma$, which is mapped to metric depth through

$D = \frac{1}{a\sigma + b},$

with $(a,b)$ chosen so that $[d_t^i,\dot d_t^i]^\top$0 m for training and $[d_t^i,\dot d_t^i]^\top$1 m for evaluation (Boulahbal et al., 2023). Temporal coupling enters through attention over tokens concatenated across four frames and through recursive prediction of future feature states.

EMoTive instead represents each pixel’s trajectory over normalized time $[d_t^i,\dot d_t^i]^\top$2 by a non-uniform rational B-spline,

$[d_t^i,\dot d_t^i]^\top$3

where $[d_t^i,\dot d_t^i]^\top$4 are control points, $[d_t^i,\dot d_t^i]^\top$5 are event-adaptive weights, and $[d_t^i,\dot d_t^i]^\top$6 are degree-$[d_t^i,\dot d_t^i]^\top$7 B-spline basis functions defined on a non-uniform knot vector (Wan et al., 14 Mar 2025). Multi-temporal samples $[d_t^i,\dot d_t^i]^\top$8 then yield optical flow vectors

$[d_t^i,\dot d_t^i]^\top$9

while depth motion is derived from the ratio of depths at consecutive time points under constant-velocity pinhole assumptions (Wan et al., 14 Mar 2025).

In CylindTrack, DTM is explicitly a dynamical system over per-track depth. Each active track $\Theta$0 carries

$\Theta$1

with constant-velocity transition

$\Theta$2

and scalar measurement model

$\Theta$3

The resulting predicted depth $\Theta$4 enters the association cost through

$\Theta$5

This replaces the naive use of raw frame-to-frame depth differences (Deng et al., 29 Jun 2026).

DepTR-MOT uses a different tracking state. After association, the track state is updated in a 3D space

$\Theta$6

with detection measurement $\Theta$7. Depth is fused into matching via

$\Theta$8

where $\Theta$9 normalizes depth distance into $L$0 and $L$1 is reported to work well empirically (Deng et al., 22 Sep 2025).

The perspective-blur formulation differs again by embedding temporal trajectory in the image-formation model. For camera rotations $L$2, a point $L$3 projects as

$L$4

and the depth-dependent blur kernel is

$L$5

The ratio of blur magnitudes for two points at depths $L$6 and $L$7 satisfies

$L$8

which is the key cue used for depth estimation (Qiu et al., 9 Dec 2025).

3. Architectural patterns and information fusion

STDepthFormer implements spatio-temporal coupling through an “ST-block” that fuses spatial features across $L$9 frames at each scale by a miniature SwinTransformer. At scale $I_{t-3}, I_{t-2}, I_{t-1}, I_t$0, each frame’s feature map $I_{t-3}, I_{t-2}, I_{t-1}, I_t$1 is projected by a $I_{t-3}, I_{t-2}, I_{t-1}, I_t$2 convolution to an embedding of $I_{t-3}, I_{t-2}, I_{t-1}, I_t$3 channels and flattened into $I_{t-3}, I_{t-2}, I_{t-1}, I_t$4 tokens; the four embeddings are concatenated along the token dimension to form a sequence of length $I_{t-3}, I_{t-2}, I_{t-1}, I_t$5. Swin-Transformer windows of depth $I_{t-3}, I_{t-2}, I_{t-1}, I_t$6, embedding dimension $I_{t-3}, I_{t-2}, I_{t-1}, I_t$7, and heads $I_{t-3}, I_{t-2}, I_{t-1}, I_t$8 operate at decreasing spatial resolutions, and self-attention is computed over all $I_{t-3}, I_{t-2}, I_{t-1}, I_t$9 tokens by

$192\times 640$0

$192\times 640$1

with $192\times 640$2. The output is reshaped back to four feature maps; the current-frame output is selected, concatenated with the original $192\times 640$3, and projected back to $192\times 640$4 channels. A temporal block then recursively maps $192\times 640$5 for $192\times 640$6 (Boulahbal et al., 2023).

EMoTive replaces tokenized RGB fusion with event-specific trajectory parameterization. It first forms an Event Kymograph,

$192\times 640$7

where $192\times 640$8 is a one-dimensional triangular spatial kernel and $192\times 640$9 is a continuous Gaussian temporal kernel. The spatial and temporal axes are decoupled by accumulation along the $D_t, D_{t+1}, D_{t+3}, D_{t+5}$0 and $D_t, D_{t+1}, D_{t+3}, D_{t+5}$1 planes separately, yielding explicit fine-grained temporal evolution. A density-aware adaptation mechanism builds a 3D event density tensor $D_t, D_{t+1}, D_{t+3}, D_{t+5}$2, average-pools it to $D_t, D_{t+1}, D_{t+3}, D_{t+5}$3, selects the top-$D_t, D_{t+1}, D_{t+3}, D_{t+5}$4 time indices, adapts knots and weights, and fuses spatial cost-volume features $D_t, D_{t+1}, D_{t+3}, D_{t+5}$5, temporal cost-volumes $D_t, D_{t+1}, D_{t+3}, D_{t+5}$6, and context $D_t, D_{t+1}, D_{t+3}, D_{t+5}$7 through

$D_t, D_{t+1}, D_{t+3}, D_{t+5}$8

The fused feature then updates control points via a small GRU (Wan et al., 14 Mar 2025).

Depth-aware MOT formulations use query-based detection pipelines. DepTR-MOT extends a DETR-style detector with a parallel depth head alongside the usual box-and-class head. Using the same learnable object queries and box centers as refinement anchors, stacked depth-aware Transformer layers produce per-query embeddings that are projected to predicted instance depth $D_t, D_{t+1}, D_{t+3}, D_{t+5}$9. Because the target MOT datasets lack ground-truth depth, the method generates soft depth labels from a frozen video-depth model and SAM2 instance masks, and additionally distills dense depth features to enforce global depth consistency (Deng et al., 22 Sep 2025).

CylindTrack places DTM inside a larger panoramic pipeline. Spherical Spatio-Temporal Consistency Learning (SSTC) refines raw query-based depth predictions into temporally smoothed, geometry-aware depth measurements $\sigma$0. DTM then filters those measurements into track-level depth states, and the Topology-Aware Cylindrical Motion Model (TCMM) augments the cylindrical motion state $\sigma$1 with the depth state $\sigma$2 (Deng et al., 29 Jun 2026).

The perspective-blur method assembles a different multimodal stack: CoTracker provides sparse point trajectories, DINOv2 supplies multi-frame video features, and DeBERTa tokenizes an up-sampled sparse rotation sequence. A 6-layer transformer decoder with self-attention, cross-attention, and feed-forward blocks fuses the tokenized trajectory with concatenated video features and predicted depth, outputting a dense rotation field $\sigma$3 (Qiu et al., 9 Dec 2025).

4. Supervision, optimization, and constraint mechanisms

STDepthFormer is trained self-supervised through novel-view synthesis. For each target time $\sigma$4, source views $\sigma$5 and $\sigma$6 are warped onto the target via

$\sigma$7

The photometric reprojection loss combines SSIM and $\sigma$8 with weight $\sigma$9 and uses the minimum over source frames to handle occlusions. Auto-masking rejects static-pixel outliers, and edge-aware disparity smoothness is weighted by $D = \frac{1}{a\sigma + b},$0:

$D = \frac{1}{a\sigma + b},$1

Supervision is applied at four decoder levels, and the full objective sums photometric and smoothness terms across targets and scales. The formulation explicitly states that no extra cycle-consistency or explicit motion-forecast term is added; temporal trajectory is enforced by applying $D = \frac{1}{a\sigma + b},$2 at each predicted future time (Boulahbal et al., 2023).

EMoTive uses a multi-task objective

$D = \frac{1}{a\sigma + b},$3

where $D = \frac{1}{a\sigma + b},$4 and $D = \frac{1}{a\sigma + b},$5 are temporally weighted sums and $D = \frac{1}{a\sigma + b},$6 is a first-order temporal smoothness term on trajectory derivatives. To evaluate such trajectory modeling, the paper introduces CarlaEvent3D: 75 synthetic driving sequences, 22 125 frames at resolution $D = \frac{1}{a\sigma + b},$7, under six environmental conditions—Day, Night, Sunset, Cloudy, Foggy, and Rainy—with splits of 45 train (13 275), 15 val (4 425), and 15 test (4 425). The dataset provides simulated events, high-precision forward optical flow, relative depth maps for motion-in-depth labels, and instance segmentation for masking object boundaries (Wan et al., 14 Mar 2025).

DepTR-MOT addresses the absence of depth annotations in MOT by defining an instance soft depth label

$D = \frac{1}{a\sigma + b},$8

where $D = \frac{1}{a\sigma + b},$9 is produced by a frozen video-depth model and $(a,b)$0 by a frozen SAM2 mask model. Per-instance depth prediction $(a,b)$1 is supervised by mean-squared error, and dense depth distillation adds a cosine-alignment term between projected encoder features and teacher depth features. The total loss weights reported in ablation are $(a,b)$2, $(a,b)$3, and $(a,b)$4 (Deng et al., 22 Sep 2025).

The perspective-blur method uses supervised losses for both depth and trajectory:

$(a,b)$5

$(a,b)$6

Its depth network employs temporal window size $(a,b)$7, $(a,b)$8, $(a,b)$9, and is trained with AdamW at $[d_t^i,\dot d_t^i]^\top$00, weight decay $[d_t^i,\dot d_t^i]^\top$01, for 320 K iterations with batch size 8 (Qiu et al., 9 Dec 2025).

A related but non-depth-specific trajectory-generation framework factorizes a time-invariant latent $[d_t^i,\dot d_t^i]^\top$02 and time-variant latents $[d_t^i,\dot d_t^i]^\top$03, and imposes spatiotemporal-validity constraints through an augmented-Lagrangian penalty,

$[d_t^i,\dot d_t^i]^\top$04

Although this work concerns general trajectory generation rather than depth, it is relevant as an example of constrained temporal latent modeling with explicit validity sets $[d_t^i,\dot d_t^i]^\top$05 (Zhang et al., 2020).

5. Empirical behavior and reported results

Reported results indicate that DTM improves different aspects of performance depending on the application: multi-step depth forecasting quality, dynamic-scene motion estimation, association robustness under occlusion, or dense trajectory recovery from blurred video. The gains are task-specific rather than directly comparable across modalities (Boulahbal et al., 2023, Wan et al., 14 Mar 2025, Deng et al., 22 Sep 2025, Deng et al., 29 Jun 2026, Qiu et al., 9 Dec 2025).

Work	Benchmark	Reported result
STDepthFormer	KITTI, Eigen split	Abs Rel $[d_t^i,\dot d_t^i]^\top$06 at $[d_t^i,\dot d_t^i]^\top$07; $[d_t^i,\dot d_t^i]^\top$08 at $[d_t^i,\dot d_t^i]^\top$09
EMoTive	CarlaEvent3D	EPE $[d_t^i,\dot d_t^i]^\top$10 px, F1 $[d_t^i,\dot d_t^i]^\top$11, log-mid $[d_t^i,\dot d_t^i]^\top$12
DepTR-MOT	QuadTrack	HOTA $[d_t^i,\dot d_t^i]^\top$13, IDF1 $[d_t^i,\dot d_t^i]^\top$14, AssA $[d_t^i,\dot d_t^i]^\top$15
DepTR-MOT	DanceTrack	HOTA $[d_t^i,\dot d_t^i]^\top$16, IDF1 $[d_t^i,\dot d_t^i]^\top$17, AssA $[d_t^i,\dot d_t^i]^\top$18
CylindTrack ablation	panoramic MOT	HOTA $[d_t^i,\dot d_t^i]^\top$19 after adding DTM
Perspective-blur method	depth benchmarks	average AbsRel $[d_t^i,\dot d_t^i]^\top$20, $[d_t^i,\dot d_t^i]^\top$21

For STDepthFormer on KITTI multi-step forecasting, the reported metrics degrade gradually with horizon: at $[d_t^i,\dot d_t^i]^\top$22, Abs Rel $[d_t^i,\dot d_t^i]^\top$23, Sq Rel $[d_t^i,\dot d_t^i]^\top$24, RMSE $[d_t^i,\dot d_t^i]^\top$25, RMSE log $[d_t^i,\dot d_t^i]^\top$26, and $[d_t^i,\dot d_t^i]^\top$27; at $[d_t^i,\dot d_t^i]^\top$28 ($[d_t^i,\dot d_t^i]^\top$29 s), Abs Rel $[d_t^i,\dot d_t^i]^\top$30, Sq Rel $[d_t^i,\dot d_t^i]^\top$31, RMSE $[d_t^i,\dot d_t^i]^\top$32, RMSE log $[d_t^i,\dot d_t^i]^\top$33, and $[d_t^i,\dot d_t^i]^\top$34. On the dynamic-object subset at $[d_t^i,\dot d_t^i]^\top$35, ManyDepth yields Abs Rel $[d_t^i,\dot d_t^i]^\top$36, the proposed model $[d_t^i,\dot d_t^i]^\top$37, and the stereo variant $[d_t^i,\dot d_t^i]^\top$38; on the static-object subset, ManyDepth reports $[d_t^i,\dot d_t^i]^\top$39 and the proposed model $[d_t^i,\dot d_t^i]^\top$40 (Boulahbal et al., 2023).

EMoTive reports, on CarlaEvent3D, a Flow EPE of $[d_t^i,\dot d_t^i]^\top$41 px, F1 $[d_t^i,\dot d_t^i]^\top$42, and log-mid $[d_t^i,\dot d_t^i]^\top$43, compared with a prior best event baseline EPE of approximately $[d_t^i,\dot d_t^i]^\top$44 and F1 of approximately $[d_t^i,\dot d_t^i]^\top$45, corresponding to an approximately $[d_t^i,\dot d_t^i]^\top$46 reduction in EPE, $[d_t^i,\dot d_t^i]^\top$47 drop in F1, and $[d_t^i,\dot d_t^i]^\top$48 drop in log-mid. On DSEC, it reports Flow EPE $[d_t^i,\dot d_t^i]^\top$49 px, F1 $[d_t^i,\dot d_t^i]^\top$50, log-mid $[d_t^i,\dot d_t^i]^\top$51, and scene-flow 3D-EPE $[d_t^i,\dot d_t^i]^\top$52 cm with Acc@5cm $[d_t^i,\dot d_t^i]^\top$53. The model size is 5.6 M parameters with 40 ms inference per 100 ms event window (Wan et al., 14 Mar 2025).

DepTR-MOT improves over a ByteTrack + DFINE baseline on both datasets reported. On QuadTrack, HOTA rises from $[d_t^i,\dot d_t^i]^\top$54 to $[d_t^i,\dot d_t^i]^\top$55, IDF1 from $[d_t^i,\dot d_t^i]^\top$56 to $[d_t^i,\dot d_t^i]^\top$57, MOTA from $[d_t^i,\dot d_t^i]^\top$58 to approximately $[d_t^i,\dot d_t^i]^\top$59, and AssA from $[d_t^i,\dot d_t^i]^\top$60 to $[d_t^i,\dot d_t^i]^\top$61, while FPS changes from $[d_t^i,\dot d_t^i]^\top$62 to $[d_t^i,\dot d_t^i]^\top$63. On DanceTrack, HOTA rises from $[d_t^i,\dot d_t^i]^\top$64 to $[d_t^i,\dot d_t^i]^\top$65, IDF1 from $[d_t^i,\dot d_t^i]^\top$66 to $[d_t^i,\dot d_t^i]^\top$67, MOTA from $[d_t^i,\dot d_t^i]^\top$68 to $[d_t^i,\dot d_t^i]^\top$69, and AssA from $[d_t^i,\dot d_t^i]^\top$70 to $[d_t^i,\dot d_t^i]^\top$71 (Deng et al., 22 Sep 2025).

CylindTrack isolates the effect of DTM in an ablation: a “Depth only” variant without DTM yields HOTA $[d_t^i,\dot d_t^i]^\top$72, IDF1 $[d_t^i,\dot d_t^i]^\top$73, and AssA $[d_t^i,\dot d_t^i]^\top$74, whereas adding DTM raises them to HOTA $[d_t^i,\dot d_t^i]^\top$75, IDF1 $[d_t^i,\dot d_t^i]^\top$76, and AssA $[d_t^i,\dot d_t^i]^\top$77. The paper attributes this to temporally filtering depth so that it becomes a more stable association cue (Deng et al., 29 Jun 2026).

The perspective-blur method reports average depth performance of AbsRel $[d_t^i,\dot d_t^i]^\top$78 and $[d_t^i,\dot d_t^i]^\top$79 on NYU, SKYScenes, and KITTI. For trajectory estimation, the inter-frame sparse trajectory achieves AbsRel $[d_t^i,\dot d_t^i]^\top$80, $[d_t^i,\dot d_t^i]^\top$81, and $[d_t^i,\dot d_t^i]^\top$82, while dense reconstruction at 15 samples per frame yields AbsRel $[d_t^i,\dot d_t^i]^\top$83, $[d_t^i,\dot d_t^i]^\top$84, and $[d_t^i,\dot d_t^i]^\top$85, and at 30 samples per frame yields AbsRel $[d_t^i,\dot d_t^i]^\top$86, $[d_t^i,\dot d_t^i]^\top$87, and $[d_t^i,\dot d_t^i]^\top$88. Ablation shows that removing Cross-Window degrades AbsRel from $[d_t^i,\dot d_t^i]^\top$89 to $[d_t^i,\dot d_t^i]^\top$90 (Qiu et al., 9 Dec 2025).

6. Common misconceptions, limitations, and extensions

A common misconception is that depth-aware temporal modeling is equivalent to simply attaching a per-frame depth scalar to each detection. The panoramic tracking literature explicitly argues against this: using raw monocular depth $[d_t^i,\dot d_t^i]^\top$91 directly in association yields only modest gains and suffers identity “jumps” when depth estimates spike, whereas DTM treats depth as a one-dimensional dynamic state that evolves smoothly along each trajectory (Deng et al., 29 Jun 2026). A related misconception is that explicit motion-forecast losses are always required. STDepthFormer states the opposite: no extra cycle-consistency or explicit motion-forecast term is added, and motion forecasting emerges implicitly from spatio-temporal attention and photometric constraints applied at each predicted future time (Boulahbal et al., 2023).

Another misconception is that increasing stochastic expressivity necessarily improves temporal depth prediction. STDepthFormer’s ablation replacing the deterministic state predictor with a VAE causes the model to collapse to one mode and worsen performance, with AbsRel $[d_t^i,\dot d_t^i]^\top$92. Removing weight sharing in the recursive state predictor also degrades AbsRel from $[d_t^i,\dot d_t^i]^\top$93 to approximately $[d_t^i,\dot d_t^i]^\top$94, indicating that parameter sharing improves generalization across forecast steps (Boulahbal et al., 2023).

The surveyed literature also reveals representation-specific limitations. DepTR-MOT requires foundation model-based supervision because existing MOT datasets largely lack depth annotations, and its depth head depends on frozen video-depth and segmentation teachers during training (Deng et al., 22 Sep 2025). EMoTive is motivated by the observation that depth variation induces spatio-temporal motion inconsistencies that disrupt assumptions of local spatial or temporal motion smoothness in previous motion-estimation frameworks, which is precisely why it adopts event-guided non-uniform parametric curves instead of uniform motion models (Wan et al., 14 Mar 2025). The perspective-blur method relies on small-rotation camera dynamics and a depth-position-dependent blur mechanism, so its DTM formulation is tied to that image-formation regime (Qiu et al., 9 Dec 2025).

Several extensions are explicitly proposed. For STDepthFormer, suggested directions include plugging in a genuine multi-hypothesis decoder such as normalizing flows to output uncertainty, incorporating semantic-attention to better disentangle object motions, and applying the same ST-block design to optical-flow or scene-flow forecasting for robotics and AV planning contexts (Boulahbal et al., 2023). More broadly, the diversity of formulations surveyed here suggests that DTM remains domain-specific: in some settings it is primarily a filtering problem, in others a sequence-prediction problem, and in others a geometric inverse problem.