---
title: Out-of-Sight Trajectory (OST) Inference
url: https://www.emergentmind.com/topics/out-of-sight-trajectory-ost
type: topic
---

# Out-of-Sight Trajectory (OST) Inference

Out-of-Sight Trajectory (OST) denotes a class of inference, tracking, and planning problems in which an object or agent is not visible to the camera, yet its state, future motion, or spatial relation must still be inferred from incomplete evidence. In the explicit trajectory-prediction formulation introduced as OOSTraj and later extended under the OST label, the task is to predict the noise-free visual trajectory of an out-of-sight agent from noisy sensor observations, using a learned vision-positioning mapping and a denoising stage before future forecasting [2404.02227], [2509.15219]. Closely related formulations study persistent 3D object localization after objects leave view, cross-frame spatial reasoning when queried objects are never co-visible, and video world models that must preserve identity and motion continuity through exit-entry events [2404.05072], [2505.24257], [2603.25716].

## 1. Terminology and scope

The literature uses several labels for closely related problems. OOSTraj defines **Out-Of-Sight Trajectory prediction** as the task of predicting the future visual trajectory of an agent that is not visible to the camera, using only that agent’s noisy sensor or localization trajectory and contextual information from other visible agents [2404.02227]. A later extension uses **Out-of-Sight Trajectory (OST)** directly and broadens the scope from out-of-sight pedestrian trajectory prediction to both pedestrians and vehicles, with applications to autonomous driving, robotics, surveillance, and virtual reality [2509.15219].

Other papers study adjacent formulations without using the OST label. **Out of Sight, Not Out of Mind (OSNOM)** focuses on maintaining the knowledge of where all objects are, as they are moved about and even when absent from the egocentric video stream, with an emphasis on active objects manipulated by the camera wearer [2404.05072]. **Disjoint-3DQA** studies egocentric spatial reasoning when two queried objects are not visible in the same frame anywhere in the video segment, so the model must integrate observations across time and reconstruct enough scene geometry to reason about an unseen object [2505.24257]. **Hybrid Memory** in video world models targets the failure mode in which dynamic subjects hide out of sight and later re-emerge, yet current systems produce frozen, distorted, or vanishing subjects [2603.25716].

This suggests that OST is best understood as a family of problems organized around hidden-state persistence under incomplete visibility. The common thread is not a single sensor stack or benchmark protocol, but the requirement that inference continue coherently when direct visual evidence is missing.

## 2. Formal problem formulations

In the canonical OOSTraj formulation, the two coupled objectives for an out-of-sight agent are to denoise its noisy sensor trajectory and to forecast its future visual trajectory in camera coordinates [2404.02227]. The denoising encoder is written as
$$
\hat{S}_{P_o}^{t_s:t_e} = \mathbf{E}_{denoise}(S_{P_o}^{t_s:t_e}),
$$
and the world-to-camera mapping is defined by
$$
[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.
$$
The projected visual trajectory is then
$$
V_{P_o}^{t_s:t_e} = M^{t_s:t_e} \cdot \hat{S}_{P_o}^{t_s:t_e},
$$
with denoising supervised indirectly in the visual domain through
$$
\mathcal{L}_{Denoise} = \mathcal{L}_2(V_{P_o}^{t_s:t_e}, \overline{V}_{P_o}^{t_s:t_e}).
$$
The later OST extension preserves this sensor-to-visual structure while reframing it as prediction of the noise-free visual trajectories of out-of-sight objects using noisy sensor data [2509.15219].

OSNOM formulates a different but closely related problem. Input observations are
$$
o_n = (f_n, m_n),
$$
where \(f_n\) is the frame index and \(m_n\) is a semantic-free 2D mask of an active object. After lifting to world coordinates, each observation becomes
$$
w_n = (f_n, l_n, v_n),
$$
with 3D location \(l_n \in \mathbb{R}^3\) and appearance descriptor \(v_n\). The defining persistence rule is
$$
\mathcal{U}(\mathcal{T}^j, \varnothing, t) \rightarrow \mathcal{T}^j_t = \mathcal{T}^j_{(t-1)},
$$
so out-of-sight handling is implemented by keeping the object’s last matched 3D world location and appearance memory unchanged until new evidence appears [2404.05072].

Disjoint-3DQA formalizes out-of-sight reasoning as non-co-visibility. For object pair \((A,B)\), visibility spans are
$$
T_A = \{t \mid A \in O_t\}, \qquad T_B = \{t \mid B \in O_t\},
$$
and the pair is included only if
$$
T_A \cap T_B = \emptyset.
$$
The benchmark therefore forces reasoning over a shared 3D layout from separated glimpses rather than direct frame-local comparison [2505.24257].

## 3. Methodological paradigms

The direct OST/OOSTraj pipeline is organized around four modules: a Mobile Denoising Encoder or Sensor Denoising Encoder, a Camera Parameters Estimator or Mapping Parameters Estimator, a Visual Positioning Projection module, and an Out-of-Sight Prediction Decoder [2404.02227], [2509.15219]. Its central claim is that clean sensor-domain supervision is unavailable, so denoising must be learned indirectly by projecting denoised sensor trajectories into the visual domain and supervising them there. The journal extension describes the Vision-Positioning Denoising module as the main technical contribution and states that it is the first work to integrate vision-positioning projection for denoising noisy sensor trajectories of out-of-sight agents [2509.15219].

OSNOM uses a markedly different paradigm: **Lift, Match, and Keep**. The method lifts partial 2D observations to 3D world coordinates using camera pose, intrinsics, aligned monocular depth, and a reconstructed scene mesh; matches observations to tracks using 3D location consistency and visual appearance similarity; and keeps tracks alive while unseen by preserving last known location and rolling appearance memory [2404.05072]. The representation is therefore piecewise-constant in hidden intervals rather than an explicit hidden-dynamics model.

Video world models introduce a third paradigm. Hybrid Memory requires models to act simultaneously as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals [2603.25716]. HyDRA implements this through a 3D-convolution-based Memory Tokenizer, explicit camera injection,
$$
H_{out} = H_{in} + \mathcal{E}_{cam}(c_{cam}),
$$
and Dynamic Retrieval Attention over motion-aware memory tokens. The training objective is the flow-matching loss
$$
\mathcal{L}_{\theta}=\mathbb{E}_{z_0,z_1,t}\left\|u(z_t,t;\theta)-v_t\right\|^2.
$$
This suggests a shift from explicit trajectory decoding to retrieval-conditioned latent continuity.

A plausible implication is that OST research now spans at least three methodological families: cross-modal denoising and projection, persistent world-coordinate memory, and retrieval-based latent continuity.

## 4. Observation, monitoring, and planning under blind spots

Some work addresses OST-like conditions not by direct hidden-target prediction, but by monitoring or planning around incomplete visibility. In autonomous driving trajectory prediction, runtime out-of-distribution awareness is formulated as quickest change-point detection over scalar prediction-error sequences \(x_t \in \{\text{ADE}_t,\text{FDE}_t,\text{RMSE}_t\}\), with CUSUM Mix reported as the strongest variant [2409.17277]. This is not direct hidden-future generation. Rather, it is a monitoring layer that detects when a predictor has entered a regime where its realized error statistics no longer match nominal behavior. An important caveat is explicit in the formulation: ADE, FDE, and RMSE require realized future ground truth.

Observation-aware planning provides a complementary response. SPOT, defined there as **Sensing-augmented Planning via Obstacle Threat modeling**, maintains a probabilistic representation of both recognized and potential obstacles in unseen space through a potential-obstacle map \(\mathcal{M}_p\) and a recognized-obstacle map \(\mathcal{M}_r\) [2510.16308]. Camera orientation is selected by
$$
\theta^\ast = \operatorname*{arg\,max}_{\theta}
\left( U_p^{R_\theta, \boldsymbol{\tau}} + U_r^{R_\theta, \boldsymbol{\tau}} \right),
$$
while the trajectory objective includes an observation term that rewards coverage of high-urgency regions. Simulation and real-world experiments show that the method detects potential dynamic obstacles 2.8 seconds earlier than baseline approaches, increases dynamic obstacle visibility by over 500\%, and enables safe navigation through cluttered, occluded environments [2510.16308].

These monitoring and planning formulations extend the OST problem from hidden-target inference to hidden-risk management. They do not solve the same estimation problem as OOSTraj, but they address the same operational failure mode: delayed response to hazards emerging from blind spots.

## 5. Benchmarks and empirical characteristics

Direct OST prediction is currently benchmarked most explicitly on **Vi-Fi** and **JRDB**. OOSTraj reports the best overall results on Vi-Fi as **SUM 27.24**, **MSE-D 13.42**, and **MSE-P 13.83**, and on JRDB as **SUM 25.51**, **MSE-D 10.52**, and **MSE-P 14.99** [2404.02227]. The later OST extension reports **Ours (Journal): 23.09 / 11.86 / 11.23** on Vi-Fi and **21.97 / 10.84 / 11.13** on JRDB, outperforming standard baselines, recent trajectory prediction models such as HIVT and AutoBots, and Kalman Filter + Transformer baselines in the reported comparisons [2509.15219].

Persistent out-of-sight object localization in egocentric video is evaluated on 100 long EPIC-KITCHENS videos averaging about 12 minutes. LMK reports that after **1 minute** **64\%** of objects are correctly positioned, after **5 minutes** **48\%**, and after **10 minutes** **37\%**; the abstract also states that after **120 seconds**, **57\%** of the objects are correctly localised by LMK, compared to **33\%** by a recent 3D method for egocentric videos and **17\%** by a general 2D tracking method [2404.05072].

Disjoint-3DQA contains **5,399 question-answer pairs**, **1,668 scenes**, and **856 unique object pairs**. Human performance is **93.96\%**; GPT-4o marked is **65.60**; Qwen 2.5-72B marked is **64.31**; and full 3D context raises GPT-4o to **83.2\%** [2505.24257]. The abstract states that models lag behind human performance by **28\%**, that accuracy declines from **60\% to 30\%** as the temporal gap widens, and that trajectories or bird’s-eye-view projections yield only marginal improvements whereas oracle 3D coordinates give a substantial **20\%** performance increase [2505.24257].

HM-World contains **59,225** clips, **17** scenes, and **49** subjects. On the reported test set, HyDRA achieves **20.357** PSNR, **0.606** SSIM, **0.289** LPIPS, **0.827** \(\text{DSC}_{ctx}\), **0.849** \(\text{DSC}_{GT}\), **0.926** Subject Consistency, and **0.932** Background Consistency [2603.25716]. The abstract summarizes the result as significant improvement over state-of-the-art approaches in both dynamic subject consistency and overall generation quality.

## 6. Conceptual boundaries, ambiguities, and open questions

A central boundary in the literature concerns what exactly remains hidden. In Disjoint-3DQA, the objects are static but the camera is moving, so the “trajectory” signal is primarily the egocentric camera path rather than object motion [2505.24257]. In OSNOM, hidden intervals are handled by a deterministic persistence model rather than continuous hidden dynamics [2404.05072]. In direct OST/OOSTraj, the target is completely out of sight and only noisy localization-like signals are available, but the method assumes access to visible agents with paired visual and sensor trajectories for camera-parameter estimation [2404.02227], [2509.15219]. In Hybrid Memory, the problem is reappearance continuity in generative video rather than explicit future-state filtering [2603.25716].

Several limitations recur across these formulations. OOSTraj relies on additional sensing such as GPS, odometry, smartphone-carried localization signals, or 3D box centers, and constructs out-of-sight examples by selectively obscuring one pedestrian in each sequence [2404.02227]. The OST journal extension notes calibration constraints, possible degradation for objects far outside the training distance regime, and open issues in extreme noise and real-time implementation [2509.15219]. OSNOM does not model uncertainty distributions, velocity, acceleration, or explicit hidden motion priors while invisible [2404.05072]. Disjoint-3DQA is built on synthetic ASE data and does not cover moving objects with true temporal trajectories, future trajectory prediction, or object re-identification under independent motion [2505.24257]. HyDRA degrades in highly complex scenes with three or more subjects and in severe occlusions [2603.25716]. Runtime OOD monitoring based on ADE, FDE, and RMSE is sequential and online in the change-point sense, but still depends on realized future motion to instantiate the error signal [2409.17277].

The term itself is also ambiguous outside this research line. In superconducting cavity diagnostics, **OST** means **Oscillating Superleak Transducer**, a second-sound detector for superfluid helium quench localization, and has no relation to hidden-agent trajectory inference [1111.5520]. A related ambiguity appears in representation-level work on generative language models: the trajectory-based OOD detector TV Score can be read as an OST-like idea only as an interpretation, because the paper does not define or use the term “Out-of-Sight Trajectory” [2405.14039].

Taken together, the literature suggests that OST is not a single settled benchmark or model family but a developing research axis centered on continuity under missing visibility. Its most mature explicit formulation is cross-modal prediction of the visual trajectory of an out-of-sight agent from noisy sensor data [2509.15219], while its broader significance lies in the convergence of tracking, memory, spatial reasoning, world modeling, monitoring, and planning around the same technical question: how to maintain coherent state when the target, hazard, or relevant relation is no longer in view.

Source: https://www.emergentmind.com/topics/out-of-sight-trajectory-ost