OOSTraj: Out-of-Sight Trajectory Prediction

• Out-of-Sight Trajectory Prediction (OOSTraj) is a framework that predicts noise-free visual trajectories for agents lacking complete visual data by leveraging sensor denoising and geometric projection.
• It integrates a vision-positioning denoising module and a mapping parameters estimator to convert noisy sensor inputs into accurate image-plane representations.
• The pipeline combines tracking, sensor fusion, and transformer-based prediction to achieve improved performance over traditional methods in autonomous driving and robotics.

Out-of-Sight Trajectory Prediction (OOSTraj) is a task and methodological framework for predicting the noise-free visual trajectories of out-of-sight objects from noisy sensor data in settings where agents are outside the camera’s field of view, fully occluded by obstacles, or intermittently visible such that no useful visual history exists. The formulation was introduced in “OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising” (Zhang et al., 2024) and extended in “Out-of-Sight Trajectories: Tracking, Fusion, and Prediction” (Zhang et al., 18 Sep 2025), where the scope is broadened from pedestrians to pedestrians and vehicles and the pipeline integrates tracking, fusion, denoising, projection, and prediction for autonomous driving, robotics, surveillance, and virtual reality.

1. Task definition and problem setting

OOSTraj targets a safety-critical regime that conventional trajectory prediction often excludes: agents for which complete and noise-free visual observations are unavailable. In the extended formulation, let $A$ denote the set of agents observed over timestamps $t \in [t_s, t_e]$. Agents are partitioned into in-sight and out-of-sight sets, with $A_{in} = \{A_i\}$ having image-plane trajectories $V_{A_i}^{t_s:t_e}$ and $A_{out} = \{A_o\}$ having only positioning or sensor trajectories $S_{A_o}^{t_s:t_e}$ collected by non-visual sensors such as GPS, FTM, UWB, IMU, and LiDAR. The raw sensor trajectories $S$ are noisy, while visual trajectories $V$ are treated as noise-free supervisory targets during training (Zhang et al., 18 Sep 2025).

The inputs comprise noisy sensor trajectories $S_{A_n}^{t_s:t_e}$ in positioning or world coordinates, visual trajectories $V_{A_i}^{t_s:t_e}$ in pixel coordinates for in-sight agents, and camera parameters, either explicit or implicitly inferred through a Mapping Parameters Estimator (MPE) as a time-varying camera matrix embedding $t \in [t_s, t_e]$0. The outputs are denoised visual trajectories for out-of-sight agents, $t \in [t_s, t_e]$1, and future visual predictions, $t \in [t_s, t_e]$2, forecasting the next $t \in [t_s, t_e]$3 steps in camera coordinates (Zhang et al., 18 Sep 2025).

The original 2024 formulation presents the same problem in a slightly different notation. In-view agents form $t \in [t_s, t_e]$4; out-of-sight agents form $t \in [t_s, t_e]$5. Every agent has a noisy sensor trajectory $t \in [t_s, t_e]$6, while in-view agents additionally have visual trajectories $t \in [t_s, t_e]$7 in pixel coordinates. During training only, ground-truth visual trajectories of the chosen out-of-sight agents are provided for loss computation; at test time, they are not available (Zhang et al., 2024).

A common misconception is to equate OOSTraj with ordinary occlusion completion from partial visual history. The task definition is stricter: it addresses targets for which no useful visual history exists, and it therefore relies on non-visual sensing plus learned cross-modal projection rather than purely image-space extrapolation (Zhang et al., 18 Sep 2025).

2. Geometric formulation and vision-positioning denoising

The central technical idea in OOSTraj is that denoising is supervised indirectly in the visual domain. The Vision-Positioning Denoising Module (VPDM) converts noisy positioning data into the visual domain and provides unsupervised denoising losses based on reprojection consistency against visual data. Its camera model is a pinhole projection,

$t \in [t_s, t_e]$8

where $t \in [t_s, t_e]$9 is the intrinsic matrix, $A_{in} = \{A_i\}$0 are extrinsics, $A_{in} = \{A_i\}$1 is a 3D world point, $A_{in} = \{A_i\}$2 is the image point, and $A_{in} = \{A_i\}$3 is scale. OOSTraj absorbs a per-scene scale $A_{in} = \{A_i\}$4 into $A_{in} = \{A_i\}$5 and writes

$A_{in} = \{A_i\}$6

For moving ego-cameras, VPDM learns a sequence $A_{in} = \{A_i\}$7 over $A_{in} = \{A_i\}$8:

$A_{in} = \{A_i\}$9

An optional radial and tangential distortion model can be integrated when available, although the reported experiments omit explicit distortion due to dataset constraints (Zhang et al., 18 Sep 2025).

VPDM learns two mappings. The Sensor Denoising Encoder $V_{A_i}^{t_s:t_e}$0, denoted $V_{A_i}^{t_s:t_e}$1, denoises raw sensor trajectories. The Visual Positioning Projection (VPP), written $V_{A_i}^{t_s:t_e}$2, aligns denoised positioning into image coordinates. Given denoised sensor points $V_{A_i}^{t_s:t_e}$3 interpreted as $V_{A_i}^{t_s:t_e}$4, OOSTraj projects them as

$V_{A_i}^{t_s:t_e}$5

The Mapping Parameters Estimator learns camera matrices from paired in-sight trajectories:

$V_{A_i}^{t_s:t_e}$6

The principal training signal is the $V_{A_i}^{t_s:t_e}$7 reprojection loss onto visual trajectories,

$V_{A_i}^{t_s:t_e}$8

with optional $V_{A_i}^{t_s:t_e}$9, $A_{out} = \{A_o\}$0, and $A_{out} = \{A_o\}$1 terms in a composite objective

$A_{out} = \{A_o\}$2

In the reported experiments, the primary term dominates, and MPE and the denoiser are trained jointly to minimize $A_{out} = \{A_o\}$3 (Zhang et al., 18 Sep 2025).

The earlier OOSTraj paper describes the same structure under the names Mobile Denoising Encoder (MDE), Camera Parameters Estimator (CPE), Visual Positioning Projection (VPP), and Out-of-Sight Prediction Decoder (OPD). There the projection is written

$A_{out} = \{A_o\}$4

with a learned time-varying camera matrix embedding

$A_{out} = \{A_o\}$5

and denoising supervision is imposed through

$A_{out} = \{A_o\}$6

This establishes the defining OOSTraj mechanism: denoising noisy sensor trajectories without ground-truth clean sensor data by constructing supervision in the image plane (Zhang et al., 2024).

3. Pipeline architecture: tracking, association, fusion, and prediction

The extended framework explicitly situates OOSTraj inside a full tracking and fusion pipeline. For in-sight agents, any standard visual detector or tracker, including SORT, DeepSORT, or MOT methods, can produce $A_{out} = \{A_o\}$7 as per-frame center points of 2D boxes on Vi-Fi or 3D boxes projected to image on JRDB. For JRDB, 3D and 2D boxes are annotated at low temporal resolution and upsampled via linear interpolation to obtain frame-level continuity, and centers are used as trajectories (Zhang et al., 18 Sep 2025).

Sensor fusion is performed by forming paired sequences $A_{out} = \{A_o\}$8 through temporal alignment and ID association. Gating can be performed via temporal nearest-neighbor or motion-continuity checks to match sensor timestamps to visual frames. These paired sequences supervise MPE because the shared camera matrix $A_{out} = \{A_o\}$9 must explain all visible agents’ cross-modal trajectory alignment. When an out-of-sight or intermittently visible agent re-enters camera view, association can be re-validated via spatial gating against projected VPDM output to synchronize identities (Zhang et al., 18 Sep 2025).

The prediction component is the Out-of-Sight Prediction Decoder (OPD), a transformer-based sequence model that forecasts future visual trajectories conditioned on denoised visual histories produced by VPDM. Its inputs are $S_{A_o}^{t_s:t_e}$0, optionally augmented with contextual cues from projected in-sight agents, and its outputs are $S_{A_o}^{t_s:t_e}$1. Training uses

$S_{A_o}^{t_s:t_e}$2

Although common trajectory forecasting metrics such as ADE and FDE are noted, OOSTraj reports image-plane MSE-T variants tailored to denoising and prediction: MSE-D for VPDM projection reconstruction, MSE-P for OPD future predictions, and their sum, SUM (Zhang et al., 18 Sep 2025).

At the architectural level, the reported modules are lightweight sequence models. The denoiser is an FC-Transformer-FC mapping $S_{A_o}^{t_s:t_e}$3; MPE is an FC-Transformer-FC mapping paired $S_{A_o}^{t_s:t_e}$4 to $S_{A_o}^{t_s:t_e}$5; VPP is a deterministic projection $S_{A_o}^{t_s:t_e}$6; and OPD is a Transformer decoder for future prediction. Models are trained with $S_{A_o}^{t_s:t_e}$7 losses on reprojection and prediction, and visual trajectories of out-of-sight agents are used as training targets but are not available at test time (Zhang et al., 18 Sep 2025).

The algorithmic summary is correspondingly direct: build in-sight pairs, infer camera matrix embeddings, denoise out-of-sight sensor trajectories, project them into the image plane, optimize reprojection loss, and then forecast future visual motion. This makes OOSTraj a cohesive pipeline rather than a standalone predictor.

4. Datasets, evaluation protocol, and empirical performance

The principal benchmarks are Vi-Fi and JRDB. Vi-Fi is a multimodal dataset combining wireless FTM, IMU, noisy GPS, and RGB-D camera data; in OOSTraj, noisy GPS is used as $S_{A_o}^{t_s:t_e}$8 and image points as $S_{A_o}^{t_s:t_e}$9, and out-of-sight is simulated by hiding one pedestrian per sequence. JRDB is egocentric robot data with 360° RGB, five cameras with $S$0 FOV each, and LiDAR; OOSTraj uses interpolated 3D box centers as sensor trajectories and image points as visual trajectories, and one pedestrian is randomly obscured to simulate out-of-sight (Zhang et al., 18 Sep 2025).

The standard protocol observes 100 timestamps of out-of-sight sensor trajectories and predicts the next 100 timestamps in the visual modality. In-sight agent pairs over the same 100 timestamps supervise MPE. Evaluation is carried out in pixels using MSE-D, MSE-P, and SUM. In the original version, OOSTraj was compared with ViTag and vanilla LSTM, GRU, RNN, and Transformer baselines; in the extended version, the benchmark is broadened to Kalman filtering and modern predictors such as HIVT and AutoBots (Zhang et al., 2024).

Setting	Vi-Fi	JRDB
OOSTraj Journal Version	SUM 23.09, MSE-D 11.86, MSE-P 11.23	SUM 21.97, MSE-D 10.84, MSE-P 11.13
Transformer	SUM 28.33, MSE-D 14.26, MSE-P 14.08	SUM 33.37, MSE-D 16.71, MSE-P 16.66
CVPR OOSTraj	SUM 27.24	SUM 25.51

The selected results indicate that the journal version surpasses the Transformer baseline by $S$1 in Vi-Fi SUM and by $S$2 in JRDB SUM, with corresponding improvements in both denoising and prediction components. Relative to the CVPR version, SUM improves from $S$3 on Vi-Fi and from $S$4 on JRDB. Against adapted SOTA prediction baselines, OOSTraj Journal Version also outperforms HIVT on Vi-Fi, where HIVT records SUM $S$5, and AutoBots, where the reported SUM is $S$6 (Zhang et al., 18 Sep 2025).

The plug-and-play study is a distinctive empirical claim. Adding the vision-positioning denoising module to standard predictors drastically reduces error. One reported example is Vi-Fi LSTM two-stage SUM $S$7 versus LSTM+VPD SUM $S$8, a $S$9 reduction. The earlier paper similarly reports JRDB gains such as Vanilla Transformer “+2 Stage” SUM $V$0 versus “+VPD (Ours)” SUM $V$1 and Vanilla RNN “+2 Stage” SUM $V$2 versus “+VPD (Ours)” SUM $V$3 (Zhang et al., 18 Sep 2025).

A further comparison is made against classical linear-Gaussian filtering. On Vi-Fi, “Kalman filter + Transformer” reports SUM $V$4, MSE-D $V$5, and MSE-P $V$6, whereas VPDM + Transformer reports SUM $V$7, MSE-D $V$8, and MSE-P $V$9. The stated explanation is that VPDM uses camera geometry constraints through $S_{A_n}^{t_s:t_e}$0, exploits accurate visual supervision of in-sight agents and training-time visual trajectories of out-of-sight agents, and learns a time-varying mapping $S_{A_n}^{t_s:t_e}$1 jointly from multi-agent data, whereas Kalman filtering lacks direct access to camera constraints and assumes a fixed linear dynamics and measurement model (Zhang et al., 18 Sep 2025).

Ablation studies consistently show that removing SDE or MDE, MPE or CPE, VPP, or OPD harms performance, confirming each component’s necessity in both the original and extended formulations (Zhang et al., 18 Sep 2025).

5. Relation to adjacent formulations in the OOSTraj literature

OOSTraj sits within a broader line of work on prediction under occlusion and modality incompleteness, but its specific formulation is geometry-aware projection of noisy non-visual trajectories into camera coordinates. One adjacent approach is “Layout Sequence Prediction From Noisy Mobile Modality” (Zhang et al., 2023), which introduces LTrajDiff, a denoising diffusion model that predicts pedestrian layout sequences under out-of-sight and occlusion constraints by conditioning on noisy mobile phone sensor data and sparse visual inputs. Its state is a layout sequence $S_{A_n}^{t_s:t_e}$2 with per-step representation

$S_{A_n}^{t_s:t_e}$3

and missing visual observations are encoded through a binary mask $S_{A_n}^{t_s:t_e}$4. The learning objective is

$S_{A_n}^{t_s:t_e}$5

LTrajDiff uses a Random Mask Strategy, a Siamese Masked Encoding Module, a Modality Fusion Module, and a coarse-to-fine denoising diffusion process. On randomly obstructed sequences it reports H3D MSE-T $S_{A_n}^{t_s:t_e}$6 and Vi-Fi MSE-T $S_{A_n}^{t_s:t_e}$7 with IoU-D $S_{A_n}^{t_s:t_e}$8, and on Vi-Fi Phase-II extremely short inputs it reports MSE-T $S_{A_n}^{t_s:t_e}$9 (Zhang et al., 2023).

A second adjacent formulation is “Navigation under uncertainty: Trajectory prediction and occlusion reasoning with switching dynamical systems” (Wei et al., 2024). There, out-of-sight prediction is posed as maintaining a probability distribution over current and future states of potentially occluded agents with an explicit binary occlusion variable $V_{A_i}^{t_s:t_e}$0 and, for some settings, an existence variable $V_{A_i}^{t_s:t_e}$1. The structured probabilistic model uses switching dynamical systems or recurrent SLDS, mode-conditioned Kalman filtering, deterministic line-of-sight geometry or visibility polygons for occlusion reasoning, and variational inference for slot-based multi-object occlusion. The reported Waymo experiments indicate that rSLDS improved ADE and normalized calibration error relative to GMM baselines for vehicles, while pedestrians were often near-constant velocity, making cGMM competitive (Wei et al., 2024).

These neighboring lines clarify OOSTraj’s position. LTrajDiff emphasizes full layout sequence generation with depth and diffusion-based multimodal reconstruction from sparse vision and mobile signals. The SLDS formulation emphasizes uncertainty maintenance, calibration, and explicit occlusion reasoning. By contrast, OOSTraj’s distinctive contribution is the integration of vision-positioning projection for unsupervised denoising of out-of-sight sensor trajectories, using camera geometry and in-sight multi-agent pairs to learn time-varying camera matrices without explicit calibration (Zhang et al., 18 Sep 2025).

6. Practical deployment, limitations, and research directions

The practical replication protocol for OOSTraj is explicit. Data preparation extracts noisy GPS or LiDAR-derived centers as sensor trajectories and image points as visual trajectories, hides one agent to simulate out-of-sight, synchronizes $V_{A_i}^{t_s:t_e}$2 and $V_{A_i}^{t_s:t_e}$3 for in-sight agents by timestamp and ID, trains SDE and MPE jointly by minimizing reprojection loss on out-of-sight agents using visual targets only during training, projects denoised sensor trajectories into the image plane at test time using MPE inferred from in-sight agents, trains OPD on denoised visual histories, and evaluates with MSE-D, MSE-P, and SUM. The code and preprocessed datasets are released at github.com/Hai-chao-Zhang/OST for the extended work and at github.com/Hai-chao-Zhang/OOSTraj for the earlier version (Zhang et al., 18 Sep 2025).

Within an autonomy stack, VPDM is described as a projection-denoising layer between non-visual localization and perception or planning stacks. Downstream motion forecasting and planning can consume $V_{A_i}^{t_s:t_e}$4 in camera coordinates or convert to world frame using inverse camera mapping if needed. The safety guidance is conservative: use uncertainty-aware planning when agents reappear from blind spots, validate MPE stability in dynamic camera motion, and include outlier rejection for spurious sensor readings (Zhang et al., 18 Sep 2025).

The limitations are equally explicit. The 2024 paper notes that the method requires in-view agents to estimate camera parameters, treats image-plane trajectories as noise-free for supervision, simplifies the camera model by possibly fixing height and omitting detailed calibration such as lens distortion, and does not model uncertainty or multimodal futures because it uses deterministic $V_{A_i}^{t_s:t_e}$5 objectives (Zhang et al., 2024). The 2025 extension adds calibration constraints under variable intrinsics and extrinsics, performance degradation at extreme distances, challenges in multi-camera setups, and difficulty under very long durations without any in-sight references (Zhang et al., 18 Sep 2025).

The future directions in the extended paper are correspondingly geometric and probabilistic: jointly learning $V_{A_i}^{t_s:t_e}$6 for multi-camera systems, incorporating ground-plane constraints, map topology, and semantic lanes or sidewalks, active sensing to select informative in-sight agents for stabilizing camera embedding estimation, and outputting multimodal distributions or heteroscedastic variances for denoised and future trajectories. A plausible implication is that subsequent OOSTraj research may converge toward hybrid systems that retain OOSTraj’s explicit cross-modal projection while absorbing the uncertainty modeling emphasized by SLDS-style occlusion reasoning and the generative flexibility of diffusion-based layout prediction (Zhang et al., 18 Sep 2025).