Out-of-Sight Trajectory (OST) Inference

• Out-of-Sight Trajectory (OST) is defined as a class of problems that predict the noise-free visual trajectory of an unseen agent using noisy sensor inputs and contextual mapping.
• It employs cross-modal denoising, spatial reasoning, and projection modules to maintain coherent tracking despite incomplete visibility.
• Applications span autonomous driving, robotics, and surveillance, with benchmark results on datasets like Vi-Fi and JRDB demonstrating its predictive prowess.

Out-of-Sight Trajectory (OST) denotes a class of inference, tracking, and planning problems in which an object or agent is not visible to the camera, yet its state, future motion, or spatial relation must still be inferred from incomplete evidence. In the explicit trajectory-prediction formulation introduced as OOSTraj and later extended under the OST label, the task is to predict the noise-free visual trajectory of an out-of-sight agent from noisy sensor observations, using a learned vision-positioning mapping and a denoising stage before future forecasting (Zhang et al., 2024, Zhang et al., 18 Sep 2025). Closely related formulations study persistent 3D object localization after objects leave view, cross-frame spatial reasoning when queried objects are never co-visible, and video world models that must preserve identity and motion continuity through exit-entry events (Plizzari et al., 2024, Ravi et al., 30 May 2025, Chen et al., 26 Mar 2026).

1. Terminology and scope

The literature uses several labels for closely related problems. OOSTraj defines Out-Of-Sight Trajectory prediction as the task of predicting the future visual trajectory of an agent that is not visible to the camera, using only that agent’s noisy sensor or localization trajectory and contextual information from other visible agents (Zhang et al., 2024). A later extension uses Out-of-Sight Trajectory (OST) directly and broadens the scope from out-of-sight pedestrian trajectory prediction to both pedestrians and vehicles, with applications to autonomous driving, robotics, surveillance, and virtual reality (Zhang et al., 18 Sep 2025).

Other papers study adjacent formulations without using the OST label. Out of Sight, Not Out of Mind (OSNOM) focuses on maintaining the knowledge of where all objects are, as they are moved about and even when absent from the egocentric video stream, with an emphasis on active objects manipulated by the camera wearer (Plizzari et al., 2024). Disjoint-3DQA studies egocentric spatial reasoning when two queried objects are not visible in the same frame anywhere in the video segment, so the model must integrate observations across time and reconstruct enough scene geometry to reason about an unseen object (Ravi et al., 30 May 2025). Hybrid Memory in video world models targets the failure mode in which dynamic subjects hide out of sight and later re-emerge, yet current systems produce frozen, distorted, or vanishing subjects (Chen et al., 26 Mar 2026).

This suggests that OST is best understood as a family of problems organized around hidden-state persistence under incomplete visibility. The common thread is not a single sensor stack or benchmark protocol, but the requirement that inference continue coherently when direct visual evidence is missing.

2. Formal problem formulations

In the canonical OOSTraj formulation, the two coupled objectives for an out-of-sight agent are to denoise its noisy sensor trajectory and to forecast its future visual trajectory in camera coordinates (Zhang et al., 2024). The denoising encoder is written as

$$\hat{S}_{P_o}^{t_s:t_e} = \mathbf{E}_{denoise}(S_{P_o}^{t_s:t_e}),$$

and the world-to-camera mapping is defined by

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$

The projected visual trajectory is then

$$V_{P_o}^{t_s:t_e} = M^{t_s:t_e} \cdot \hat{S}_{P_o}^{t_s:t_e},$$

with denoising supervised indirectly in the visual domain through

$$\mathcal{L}_{Denoise} = \mathcal{L}_2(V_{P_o}^{t_s:t_e}, \overline{V}_{P_o}^{t_s:t_e}).$$

The later OST extension preserves this sensor-to-visual structure while reframing it as prediction of the noise-free visual trajectories of out-of-sight objects using noisy sensor data (Zhang et al., 18 Sep 2025).

OSNOM formulates a different but closely related problem. Input observations are

$o_n = (f_n, m_n),$

where $f_n$ is the frame index and $m_n$ is a semantic-free 2D mask of an active object. After lifting to world coordinates, each observation becomes

$w_n = (f_n, l_n, v_n),$

with 3D location $l_n \in \mathbb{R}^3$ and appearance descriptor $v_n$. The defining persistence rule is

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$0

so out-of-sight handling is implemented by keeping the object’s last matched 3D world location and appearance memory unchanged until new evidence appears (Plizzari et al., 2024).

Disjoint-3DQA formalizes out-of-sight reasoning as non-co-visibility. For object pair $$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$1, visibility spans are

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$2

and the pair is included only if

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$3

The benchmark therefore forces reasoning over a shared 3D layout from separated glimpses rather than direct frame-local comparison (Ravi et al., 30 May 2025).

3. Methodological paradigms

The direct OST/OOSTraj pipeline is organized around four modules: a Mobile Denoising Encoder or Sensor Denoising Encoder, a Camera Parameters Estimator or Mapping Parameters Estimator, a Visual Positioning Projection module, and an Out-of-Sight Prediction Decoder (Zhang et al., 2024, Zhang et al., 18 Sep 2025). Its central claim is that clean sensor-domain supervision is unavailable, so denoising must be learned indirectly by projecting denoised sensor trajectories into the visual domain and supervising them there. The journal extension describes the Vision-Positioning Denoising module as the main technical contribution and states that it is the first work to integrate vision-positioning projection for denoising noisy sensor trajectories of out-of-sight agents (Zhang et al., 18 Sep 2025).

OSNOM uses a markedly different paradigm: Lift, Match, and Keep. The method lifts partial 2D observations to 3D world coordinates using camera pose, intrinsics, aligned monocular depth, and a reconstructed scene mesh; matches observations to tracks using 3D location consistency and visual appearance similarity; and keeps tracks alive while unseen by preserving last known location and rolling appearance memory (Plizzari et al., 2024). The representation is therefore piecewise-constant in hidden intervals rather than an explicit hidden-dynamics model.

Video world models introduce a third paradigm. Hybrid Memory requires models to act simultaneously as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals (Chen et al., 26 Mar 2026). HyDRA implements this through a 3D-convolution-based Memory Tokenizer, explicit camera injection,

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$4

and Dynamic Retrieval Attention over motion-aware memory tokens. The training objective is the flow-matching loss

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$5

This suggests a shift from explicit trajectory decoding to retrieval-conditioned latent continuity.

A plausible implication is that OST research now spans at least three methodological families: cross-modal denoising and projection, persistent world-coordinate memory, and retrieval-based latent continuity.

4. Observation, monitoring, and planning under blind spots

Some work addresses OST-like conditions not by direct hidden-target prediction, but by monitoring or planning around incomplete visibility. In autonomous driving trajectory prediction, runtime out-of-distribution awareness is formulated as quickest change-point detection over scalar prediction-error sequences $$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$6, with CUSUM Mix reported as the strongest variant (Guo et al., 2024). This is not direct hidden-future generation. Rather, it is a monitoring layer that detects when a predictor has entered a regime where its realized error statistics no longer match nominal behavior. An important caveat is explicit in the formulation: ADE, FDE, and RMSE require realized future ground truth.

Observation-aware planning provides a complementary response. SPOT, defined there as Sensing-augmented Planning via Obstacle Threat modeling, maintains a probabilistic representation of both recognized and potential obstacles in unseen space through a potential-obstacle map $$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$7 and a recognized-obstacle map $$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$8 (Zhang et al., 18 Oct 2025). Camera orientation is selected by

$$[p, 1]^T = w \cdot K \cdot R_t \cdot [P, 1]^T, \qquad M = w \cdot K \cdot R_t.$$9

while the trajectory objective includes an observation term that rewards coverage of high-urgency regions. Simulation and real-world experiments show that the method detects potential dynamic obstacles 2.8 seconds earlier than baseline approaches, increases dynamic obstacle visibility by over 500\%, and enables safe navigation through cluttered, occluded environments (Zhang et al., 18 Oct 2025).

These monitoring and planning formulations extend the OST problem from hidden-target inference to hidden-risk management. They do not solve the same estimation problem as OOSTraj, but they address the same operational failure mode: delayed response to hazards emerging from blind spots.

5. Benchmarks and empirical characteristics

Direct OST prediction is currently benchmarked most explicitly on Vi-Fi and JRDB. OOSTraj reports the best overall results on Vi-Fi as SUM 27.24, MSE-D 13.42, and MSE-P 13.83, and on JRDB as SUM 25.51, MSE-D 10.52, and MSE-P 14.99 (Zhang et al., 2024). The later OST extension reports Ours (Journal): 23.09 / 11.86 / 11.23 on Vi-Fi and 21.97 / 10.84 / 11.13 on JRDB, outperforming standard baselines, recent trajectory prediction models such as HIVT and AutoBots, and Kalman Filter + Transformer baselines in the reported comparisons (Zhang et al., 18 Sep 2025).

Persistent out-of-sight object localization in egocentric video is evaluated on 100 long EPIC-KITCHENS videos averaging about 12 minutes. LMK reports that after 1 minute 64\% of objects are correctly positioned, after 5 minutes 48\%, and after 10 minutes 37\%; the abstract also states that after 120 seconds, 57\% of the objects are correctly localised by LMK, compared to 33\% by a recent 3D method for egocentric videos and 17\% by a general 2D tracking method (Plizzari et al., 2024).

Disjoint-3DQA contains 5,399 question-answer pairs, 1,668 scenes, and 856 unique object pairs. Human performance is 93.96\%; GPT-4o marked is 65.60; Qwen 2.5-72B marked is 64.31; and full 3D context raises GPT-4o to 83.2\% (Ravi et al., 30 May 2025). The abstract states that models lag behind human performance by 28\%, that accuracy declines from 60\% to 30\% as the temporal gap widens, and that trajectories or bird’s-eye-view projections yield only marginal improvements whereas oracle 3D coordinates give a substantial 20\% performance increase (Ravi et al., 30 May 2025).

HM-World contains 59,225 clips, 17 scenes, and 49 subjects. On the reported test set, HyDRA achieves 20.357 PSNR, 0.606 SSIM, 0.289 LPIPS, 0.827 $$V_{P_o}^{t_s:t_e} = M^{t_s:t_e} \cdot \hat{S}_{P_o}^{t_s:t_e},$$0, 0.849 $$V_{P_o}^{t_s:t_e} = M^{t_s:t_e} \cdot \hat{S}_{P_o}^{t_s:t_e},$$1, 0.926 Subject Consistency, and 0.932 Background Consistency (Chen et al., 26 Mar 2026). The abstract summarizes the result as significant improvement over state-of-the-art approaches in both dynamic subject consistency and overall generation quality.

6. Conceptual boundaries, ambiguities, and open questions

A central boundary in the literature concerns what exactly remains hidden. In Disjoint-3DQA, the objects are static but the camera is moving, so the “trajectory” signal is primarily the egocentric camera path rather than object motion (Ravi et al., 30 May 2025). In OSNOM, hidden intervals are handled by a deterministic persistence model rather than continuous hidden dynamics (Plizzari et al., 2024). In direct OST/OOSTraj, the target is completely out of sight and only noisy localization-like signals are available, but the method assumes access to visible agents with paired visual and sensor trajectories for camera-parameter estimation (Zhang et al., 2024, Zhang et al., 18 Sep 2025). In Hybrid Memory, the problem is reappearance continuity in generative video rather than explicit future-state filtering (Chen et al., 26 Mar 2026).

Several limitations recur across these formulations. OOSTraj relies on additional sensing such as GPS, odometry, smartphone-carried localization signals, or 3D box centers, and constructs out-of-sight examples by selectively obscuring one pedestrian in each sequence (Zhang et al., 2024). The OST journal extension notes calibration constraints, possible degradation for objects far outside the training distance regime, and open issues in extreme noise and real-time implementation (Zhang et al., 18 Sep 2025). OSNOM does not model uncertainty distributions, velocity, acceleration, or explicit hidden motion priors while invisible (Plizzari et al., 2024). Disjoint-3DQA is built on synthetic ASE data and does not cover moving objects with true temporal trajectories, future trajectory prediction, or object re-identification under independent motion (Ravi et al., 30 May 2025). HyDRA degrades in highly complex scenes with three or more subjects and in severe occlusions (Chen et al., 26 Mar 2026). Runtime OOD monitoring based on ADE, FDE, and RMSE is sequential and online in the change-point sense, but still depends on realized future motion to instantiate the error signal (Guo et al., 2024).

The term itself is also ambiguous outside this research line. In superconducting cavity diagnostics, OST means Oscillating Superleak Transducer, a second-sound detector for superfluid helium quench localization, and has no relation to hidden-agent trajectory inference (Quadt et al., 2011). A related ambiguity appears in representation-level work on generative LLMs: the trajectory-based OOD detector TV Score can be read as an OST-like idea only as an interpretation, because the paper does not define or use the term “Out-of-Sight Trajectory” (Wang et al., 2024).

Taken together, the literature suggests that OST is not a single settled benchmark or model family but a developing research axis centered on continuity under missing visibility. Its most mature explicit formulation is cross-modal prediction of the visual trajectory of an out-of-sight agent from noisy sensor data (Zhang et al., 18 Sep 2025), while its broader significance lies in the convergence of tracking, memory, spatial reasoning, world modeling, monitoring, and planning around the same technical question: how to maintain coherent state when the target, hazard, or relevant relation is no longer in view.