---
title: Depth-Temporal Trajectory Modeling
url: https://www.emergentmind.com/topics/depth-temporal-trajectory-modeling-dtm
type: topic
---

# Depth-Temporal Trajectory Modeling

Searching arXiv for recent and directly relevant papers on “Depth-Temporal Trajectory Modeling” and related formulations.
Depth-Temporal Trajectory Modeling (DTM) denotes a family of formulations in which depth is modeled as a temporally evolving quantity rather than as an isolated frame-wise estimate. In the literature considered here, the term spans several distinct settings: future depth-sequence forecasting from RGB video, event-guided parametric trajectory fitting for 3D motion estimation, instance-level depth refinement for multi-object tracking, Kalman-filtered depth states for panoramic tracking, and joint recovery of metric depth with dense camera-rotation trajectories from perspective-based blur [2303.01196] [2503.11371] [2509.17323] [2606.30097] [2512.08627]. A related trajectory-generation line factorizes global and local latent variables under spatiotemporal-validity constraints, which does not itself model depth but is relevant to the temporal-structure side of the problem [2009.09333].

## 1. Taxonomic scope

The surveyed work does not present DTM as a single standardized algorithm. Instead, the common thread is that depth participates in temporal structure: as a predicted sequence, as a latent dynamical state, as a trajectory-conditioned geometric cue, or as a quantity coupled to image formation over time. This suggests that DTM is best understood as a modeling perspective rather than a fixed architecture.

| Work | Observation domain | DTM object |
|---|---|---|
| STDepthFormer | RGB video | forecast depth maps \(D_t, D_{t+1}, D_{t+3}, D_{t+5}\) |
| EMoTive | event stream | event-guided non-uniform rational curves for motion and depth motion |
| DepTR-MOT | MOT detection/tracking | instance-level depth integrated into matching and state update |
| CylindTrack | panoramic MOT | Kalman-filtered depth state \([d_t^i,\dot d_t^i]^\top\) |
| Perspective-based blur method | blurred monocular video | dense rotation trajectory \(\Theta\) coupled to metric depth map \(L\) |

In STDepthFormer, four consecutive RGB images \(I_{t-3}, I_{t-2}, I_{t-1}, I_t\) resized to \(192\times 640\) are mapped to four forecast depth outputs \(D_t, D_{t+1}, D_{t+3}, D_{t+5}\), with each depth produced at four pyramid levels [2303.01196]. In EMoTive, DTM refers to modeling spatio-temporal trajectories via event-guided non-uniform parametric curves, followed by multi-temporal sampling to recover optical flow and depth motion fields [2503.11371]. In DepTR-MOT and CylindTrack, DTM operates inside tracking-by-detection pipelines: the former predicts per-instance depth and fuses it into association and 3D state updates, whereas the latter promotes depth to a trajectory-level state filtered by a one-dimensional constant-velocity Kalman filter [2509.17323] [2606.30097]. In the perspective-blur setting, DTM denotes a two-stage pipeline that estimates metric depth and densifies a sparse optical trajectory into a dense rotation field [2512.08627].

## 2. State representations and mathematical formulations

A central distinction across DTM formulations is the choice of state space. STDepthFormer uses depth maps as the temporal state. Its network predicts a disparity-like activation \(\sigma\), which is mapped to metric depth through
$$
D = \frac{1}{a\sigma + b},
$$
with \((a,b)\) chosen so that \(D \in [0.1,100]\) m for training and \(D \in [0.5,100]\) m for evaluation [2303.01196]. Temporal coupling enters through attention over tokens concatenated across four frames and through recursive prediction of future feature states.

EMoTive instead represents each pixel’s trajectory over normalized time \(t\in[0,1]\) by a non-uniform rational B-spline,
$$
\mathbf{X}(t)
=
\frac{\sum_{i=0}^n N_{i,p}(t)\,w_i\,\mathbf{P}_i}
{\sum_{i=0}^n N_{i,p}(t)\,w_i},
$$
where \(\mathbf{P}_i\in\mathbb{R}^2\) are control points, \(w_i>0\) are event-adaptive weights, and \(N_{i,p}(t)\) are degree-\(p\) B-spline basis functions defined on a non-uniform knot vector [2503.11371]. Multi-temporal samples \(\{\mathbf{X}(t_k)\}_{k=1}^K\) then yield optical flow vectors
$$
\mathbf{u}_k = \mathbf{X}(t_{k+1}) - \mathbf{X}(t_k),
$$
while depth motion is derived from the ratio of depths at consecutive time points under constant-velocity pinhole assumptions [2503.11371].

In CylindTrack, DTM is explicitly a dynamical system over per-track depth. Each active track \(i\) carries
$$
\mathbf{x}_t^i = \begin{bmatrix} d_t^i \\ \dot d_t^i \end{bmatrix},
$$
with constant-velocity transition
$$
\mathbf{x}_t^i = F\,\mathbf{x}_{t-1}^i + \mathbf{w}_t,\qquad
F=\begin{bmatrix}1&1\\0&1\end{bmatrix},
$$
and scalar measurement model
$$
z_t^j = H\,\mathbf{x}_t^i + \eta_t,\qquad H=[\,1\;\;0\,].
$$
The resulting predicted depth \(\hat d_t^i\) enters the association cost through
$$
\mathcal{C}_{ij}^{\rm dep} = |\hat d_t^i - z_t^j|.
$$
This replaces the naive use of raw frame-to-frame depth differences [2606.30097].

DepTR-MOT uses a different tracking state. After association, the track state is updated in a 3D space
$$
\mathbf{x}_t = \begin{bmatrix}x_t\\y_t\\z_t\end{bmatrix},
$$
with detection measurement \(\mathbf{z}_d=(u_d,v_d,P_d)^\top\). Depth is fused into matching via
$$
D_{ij}=\eta\,|P_{t_i}-P_{d_j}|,
\qquad
C'_{ij}=\lambda\,C_{ij}+\gamma\,D_{ij},
$$
where \(\eta\) normalizes depth distance into \([0,1]\) and \(\gamma\approx 0.2\ldots 0.4\) is reported to work well empirically [2509.17323].

The perspective-blur formulation differs again by embedding temporal trajectory in the image-formation model. For camera rotations \(\omega(t)\in\mathbb{R}^3\), a point \(X=(X,Y,Z)^\top\) projects as
$$
x(t)=\pi(R(t)X),\qquad \pi([X,Y,Z]^\top)=\bigl(fX/Z,\;fY/Z\bigr)^\top,
$$
and the depth-dependent blur kernel is
$$
K(u;\omega(\cdot),Z)=\frac{1}{T_E}\int_0^{T_E}\delta\bigl(u-\delta x(t;Z)\bigr)\,dt.
$$
The ratio of blur magnitudes for two points at depths \(Z_{\rm on}\) and \(Z_{\rm off}\) satisfies
$$
\frac{\rho_{\rm off}}{\rho_{\rm on}} \propto \frac{Z_{\rm on}}{Z_{\rm off}},
$$
which is the key cue used for depth estimation [2512.08627].

## 3. Architectural patterns and information fusion

STDepthFormer implements spatio-temporal coupling through an “ST-block” that fuses spatial features across \(k=4\) frames at each scale by a miniature SwinTransformer. At scale \(s\), each frame’s feature map \(F_i^{(s)}\in\mathbb{R}^{C_s\times W_s\times H_s}\) is projected by a \(1\times1\) convolution to an embedding of \(d=96\) channels and flattened into \(N_s=W_s\cdot H_s\) tokens; the four embeddings are concatenated along the token dimension to form a sequence of length \(T_s=4N_s\). Swin-Transformer windows of depth \(2\), embedding dimension \(96\), and heads \(H_s=[3,6,12,24]\) operate at decreasing spatial resolutions, and self-attention is computed over all \(T_s\) tokens by
$$
Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V,
$$
$$
\mathrm{Attention}(X)=\mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V,
$$
with \(X\in\mathbb{R}^{T_s\times 96}\). The output is reshaped back to four feature maps; the current-frame output is selected, concatenated with the original \(F_t\), and projected back to \(C_s\) channels. A temporal block then recursively maps \(F_{t+k-1}\rightarrow F_{t+k}\) for \(k\in\{1,3,5\}\) [2303.01196].

EMoTive replaces tokenized RGB fusion with event-specific trajectory parameterization. It first forms an Event Kymograph,
$$
K_x(t,x)=\sum_{i=1}^N p_i\,k(x-x_i)\,g(t-t_i\mid \sigma),\qquad
K_y(t,y)=\sum_{i=1}^N p_i\,k(y-y_i)\,g(t-t_i\mid \sigma),
$$
where \(k(a)=\max(0,1-|a|)\) is a one-dimensional triangular spatial kernel and \(g(\Delta t\mid \sigma)=\exp(-(\Delta t/\sigma)^2)\) is a continuous Gaussian temporal kernel. The spatial and temporal axes are decoupled by accumulation along the \(x\text{--}t\) and \(y\text{--}t\) planes separately, yielding explicit fine-grained temporal evolution. A density-aware adaptation mechanism builds a 3D event density tensor \(E_s\), average-pools it to \(D_s\), selects the top-\(n\) time indices, adapts knots and weights, and fuses spatial cost-volume features \(F_s\), temporal cost-volumes \(F_t\), and context \(f_c\) through
$$
\mathcal{F}=\mathrm{Conv}[F_s\|F_t\|f_c].
$$
The fused feature then updates control points via a small GRU [2503.11371].

Depth-aware MOT formulations use query-based detection pipelines. DepTR-MOT extends a DETR-style detector with a parallel depth head alongside the usual box-and-class head. Using the same learnable object queries and box centers as refinement anchors, stacked depth-aware Transformer layers produce per-query embeddings that are projected to predicted instance depth \(P_q\). Because the target MOT datasets lack ground-truth depth, the method generates soft depth labels from a frozen video-depth model and SAM2 instance masks, and additionally distills dense depth features to enforce global depth consistency [2509.17323].

CylindTrack places DTM inside a larger panoramic pipeline. Spherical Spatio-Temporal Consistency Learning (SSTC) refines raw query-based depth predictions into temporally smoothed, geometry-aware depth measurements \(z_t^j\). DTM then filters those measurements into track-level depth states, and the Topology-Aware Cylindrical Motion Model (TCMM) augments the cylindrical motion state \((\theta_t^i,y_t^i,a_t^i,h_t^i,\dot\theta_t^i,\dot y_t^i,\dot a_t^i,\dot h_t^i)\) with the depth state \(\mathbf{x}_t^i\) [2606.30097].

The perspective-blur method assembles a different multimodal stack: CoTracker provides sparse point trajectories, DINOv2 supplies multi-frame video features, and DeBERTa tokenizes an up-sampled sparse rotation sequence. A 6-layer transformer decoder with self-attention, cross-attention, and feed-forward blocks fuses the tokenized trajectory with concatenated video features and predicted depth, outputting a dense rotation field \(\Theta_{\rm dense}\in\mathbb{R}^{S\times 3}\) [2512.08627].

## 4. Supervision, optimization, and constraint mechanisms

STDepthFormer is trained self-supervised through novel-view synthesis. For each target time \(\mathrm{tgt}\in\{0,1,3,5\}\), source views \(I_{\mathrm{tgt}-1}\) and \(I_{\mathrm{tgt}+1}\) are warped onto the target via
$$
\hat p_{\rm src} \sim K\,T_{{\rm src}\rightarrow{\rm tgt}}\,D_{\rm tgt}\,K^{-1}p_{\rm tgt}.
$$
The photometric reprojection loss combines SSIM and \(L_1\) with weight \(\alpha=0.15\) and uses the minimum over source frames to handle occlusions. Auto-masking rejects static-pixel outliers, and edge-aware disparity smoothness is weighted by \(\alpha_s=10^{-3}\):
$$
L_s(D)=\sum_p |\partial_x D(p)|e^{-|\partial_x I(p)|}+|\partial_y D(p)|e^{-|\partial_y I(p)|}.
$$
Supervision is applied at four decoder levels, and the full objective sums photometric and smoothness terms across targets and scales. The formulation explicitly states that no extra cycle-consistency or explicit motion-forecast term is added; temporal trajectory is enforced by applying \(L_{\rm ph}\) at each predicted future time [2303.01196].

EMoTive uses a multi-task objective
$$
L = L_{\rm flow}+L_{\rm depth}+\lambda L_t,
$$
where \(L_{\rm flow}\) and \(L_{\rm depth}\) are temporally weighted sums and \(L_t\) is a first-order temporal smoothness term on trajectory derivatives. To evaluate such trajectory modeling, the paper introduces CarlaEvent3D: 75 synthetic driving sequences, 22 125 frames at resolution \(320\times 960\), under six environmental conditions—Day, Night, Sunset, Cloudy, Foggy, and Rainy—with splits of 45 train (13 275), 15 val (4 425), and 15 test (4 425). The dataset provides simulated events, high-precision forward optical flow, relative depth maps for motion-in-depth labels, and instance segmentation for masking object boundaries [2503.11371].

DepTR-MOT addresses the absence of depth annotations in MOT by defining an instance soft depth label
$$
Y_i=
\frac{\sum_{(x,y)} D_t(x,y)\,M_i(x,y)}
{\sum_{(x,y)} M_i(x,y)},
$$
where \(D_t\) is produced by a frozen video-depth model and \(M_i\) by a frozen SAM2 mask model. Per-instance depth prediction \(P_i\) is supervised by mean-squared error, and dense depth distillation adds a cosine-alignment term between projected encoder features and teacher depth features. The total loss weights reported in ablation are \(\lambda_{\rm box}=2.0\), \(\lambda_{\rm reg}=1.0\), and \(\lambda_{\rm align}=0.25\) [2509.17323].

The perspective-blur method uses supervised losses for both depth and trajectory:
$$
\mathcal{L}_{\rm depth}=\|L_{\rm pred}-L_{\rm GT}\|_1+\alpha_{\rm smooth}\|\nabla L_{\rm pred}\|_1,
$$
$$
\mathcal{L}_{\rm traj}=\sum_t \|\theta_t^{\rm pred}-\theta_t^{\rm GT}\|_1.
$$
Its depth network employs temporal window size \(S=6\), \(L_{\rm SW}=6\), \(L_{\rm CW}=2\), and is trained with AdamW at \(\eta=10^{-3}\), weight decay \(5\times 10^{-5}\), for 320 K iterations with batch size 8 [2512.08627].

A related but non-depth-specific trajectory-generation framework factorizes a time-invariant latent \(w\) and time-variant latents \(z_{1:T}\), and imposes spatiotemporal-validity constraints through an augmented-Lagrangian penalty,
$$
R(\theta,\phi)=\gamma\sqrt{\mathbb{E}_{p_\theta(w,z,\Theta)}\!\left[\mathbbm{1}(g(w,z,\Theta)\notin\mathcal{C})\right]}.
$$
Although this work concerns general trajectory generation rather than depth, it is relevant as an example of constrained temporal latent modeling with explicit validity sets \(\mathcal{C}\) [2009.09333].

## 5. Empirical behavior and reported results

Reported results indicate that DTM improves different aspects of performance depending on the application: multi-step depth forecasting quality, dynamic-scene motion estimation, association robustness under occlusion, or dense trajectory recovery from blurred video. The gains are task-specific rather than directly comparable across modalities [2303.01196] [2503.11371] [2509.17323] [2606.30097] [2512.08627].

| Work | Benchmark | Reported result |
|---|---|---|
| STDepthFormer | KITTI, Eigen split | Abs Rel \(0.110\) at \(t=0\); \(0.165\) at \(t=5\) |
| EMoTive | CarlaEvent3D | EPE \(2.547\) px, F1 \(22.87\%\), log-mid \(113.59\) |
| DepTR-MOT | QuadTrack | HOTA \(27.59\), IDF1 \(28.04\), AssA \(26.55\) |
| DepTR-MOT | DanceTrack | HOTA \(44.465\), IDF1 \(47.418\), AssA \(28.62\) |
| CylindTrack ablation | panoramic MOT | HOTA \(31.026 \rightarrow 31.930\) after adding DTM |
| Perspective-blur method | depth benchmarks | average AbsRel \(0.067\), \(\delta_1=0.950\) |

For STDepthFormer on KITTI multi-step forecasting, the reported metrics degrade gradually with horizon: at \(t=0\), Abs Rel \(=0.110\), Sq Rel \(=0.805\), RMSE \(=4.678\), RMSE log \(=0.187\), and \(\delta<1.25=0.879\); at \(t=5\) (\(0.5\) s), Abs Rel \(=0.165\), Sq Rel \(=1.489\), RMSE \(=5.805\), RMSE log \(=0.245\), and \(\delta<1.25=0.792\). On the dynamic-object subset at \(t=0\), ManyDepth yields Abs Rel \(=0.192\), the proposed model \(0.178\), and the stereo variant \(0.155\); on the static-object subset, ManyDepth reports \(0.085\) and the proposed model \(0.099\) [2303.01196].

EMoTive reports, on CarlaEvent3D, a Flow EPE of \(2.547\) px, F1 \(22.87\%\), and log-mid \(113.59\), compared with a prior best event baseline EPE of approximately \(2.78\) and F1 of approximately \(24.6\), corresponding to an approximately \(8.4\%\) reduction in EPE, \(7.1\%\) drop in F1, and \(6.9\%\) drop in log-mid. On DSEC, it reports Flow EPE \(0.439\) px, F1 \(0.742\%\), log-mid \(152.72\), and scene-flow 3D-EPE \(0.205\) cm with Acc@5cm \(=25.5\%\). The model size is 5.6 M parameters with 40 ms inference per 100 ms event window [2503.11371].

DepTR-MOT improves over a ByteTrack + DFINE baseline on both datasets reported. On QuadTrack, HOTA rises from \(23.903\) to \(27.59\), IDF1 from \(23.072\) to \(28.04\), MOTA from \(-7.94\) to approximately \(-3.65\), and AssA from \(20.36\) to \(26.55\), while FPS changes from \(38\) to \(30\). On DanceTrack, HOTA rises from \(42.343\) to \(44.465\), IDF1 from \(45.922\) to \(47.418\), MOTA from \(83.352\) to \(86.032\), and AssA from \(26.64\) to \(28.62\) [2509.17323].

CylindTrack isolates the effect of DTM in an ablation: a “Depth only” variant without DTM yields HOTA \(=31.026\), IDF1 \(=34.109\), and AssA \(=29.786\), whereas adding DTM raises them to HOTA \(=31.930\), IDF1 \(=37.520\), and AssA \(=31.560\). The paper attributes this to temporally filtering depth so that it becomes a more stable association cue [2606.30097].

The perspective-blur method reports average depth performance of AbsRel \(=0.067\) and \(\delta_1=0.950\) on NYU, SKYScenes, and KITTI. For trajectory estimation, the inter-frame sparse trajectory achieves AbsRel \(=0.044\), \(e_1=0.959\), and \(e_2=1.000\), while dense reconstruction at 15 samples per frame yields AbsRel \(=0.068\), \(e_1=0.858\), and \(e_2=0.949\), and at 30 samples per frame yields AbsRel \(=0.067\), \(e_1=0.865\), and \(e_2=0.955\). Ablation shows that removing Cross-Window degrades AbsRel from \(0.067\) to \(0.091\) [2512.08627].

## 6. Common misconceptions, limitations, and extensions

A common misconception is that depth-aware temporal modeling is equivalent to simply attaching a per-frame depth scalar to each detection. The panoramic tracking literature explicitly argues against this: using raw monocular depth \(d_t^i\) directly in association yields only modest gains and suffers identity “jumps” when depth estimates spike, whereas DTM treats depth as a one-dimensional dynamic state that evolves smoothly along each trajectory [2606.30097]. A related misconception is that explicit motion-forecast losses are always required. STDepthFormer states the opposite: no extra cycle-consistency or explicit motion-forecast term is added, and motion forecasting emerges implicitly from spatio-temporal attention and photometric constraints applied at each predicted future time [2303.01196].

Another misconception is that increasing stochastic expressivity necessarily improves temporal depth prediction. STDepthFormer’s ablation replacing the deterministic state predictor with a VAE causes the model to collapse to one mode and worsen performance, with AbsRel \(\gtrsim 0.18\). Removing weight sharing in the recursive state predictor also degrades AbsRel from \(0.165\) to approximately \(0.171\), indicating that parameter sharing improves generalization across forecast steps [2303.01196].

The surveyed literature also reveals representation-specific limitations. DepTR-MOT requires foundation model-based supervision because existing MOT datasets largely lack depth annotations, and its depth head depends on frozen video-depth and segmentation teachers during training [2509.17323]. EMoTive is motivated by the observation that depth variation induces spatio-temporal motion inconsistencies that disrupt assumptions of local spatial or temporal motion smoothness in previous motion-estimation frameworks, which is precisely why it adopts event-guided non-uniform parametric curves instead of uniform motion models [2503.11371]. The perspective-blur method relies on small-rotation camera dynamics and a depth-position-dependent blur mechanism, so its DTM formulation is tied to that image-formation regime [2512.08627].

Several extensions are explicitly proposed. For STDepthFormer, suggested directions include plugging in a genuine multi-hypothesis decoder such as normalizing flows to output uncertainty, incorporating semantic-attention to better disentangle object motions, and applying the same ST-block design to optical-flow or scene-flow forecasting for robotics and AV planning contexts [2303.01196]. More broadly, the diversity of formulations surveyed here suggests that DTM remains domain-specific: in some settings it is primarily a filtering problem, in others a sequence-prediction problem, and in others a geometric inverse problem.

Source: https://www.emergentmind.com/topics/depth-temporal-trajectory-modeling-dtm