Vision-Positioning Denoising Module

• The paper introduces VPD, which refines noisy sensor trajectories into denoised latent trajectories that can be projected into the visual coordinate system.
• It leverages a Transformer-based Sensor Denoising Encoder and a Mapping Parameters Estimator to learn cross-modal calibration from visible agents.
• Empirical results on datasets like Vi-Fi and JRDB demonstrate that VPD improves denoising and downstream trajectory prediction over classical filtering methods.

The Vision-Positioning Denoising Module (VPD) is the central mechanism in the Out-of-Sight Trajectory framework for recovering a denoised visual trajectory representation of an agent that is completely outside the camera’s field of view, using only its noisy positioning trajectory together with paired visual-positioning trajectories of other visible agents (Zhang et al., 18 Sep 2025). In the earlier OOSTraj formulation, VPD is the component that makes out-of-sight trajectory prediction feasible by denoising noisy sensor observations in an unsupervised manner with respect to clean sensor labels and projecting them into the visual coordinate system before downstream prediction (Zhang et al., 2024). The module is motivated by the fact that real deployments often retain noisy localization streams such as GPS, odometry, mobile-device positioning, or 3D box centers after visual contact is lost, while direct visual correction is unavailable for the target agent.

1. Problem definition and task setting

The VPD addresses a denoising problem that is atypical in trajectory forecasting. The target agent is out of sight, meaning that during the observation interval it has no visual trajectory in the camera view, either because it is entirely outside the field of view or because it is occluded by obstacles or other vehicles (Zhang et al., 2024). At the same time, the method assumes that noisy localization trajectories remain available. The denoising task is therefore not ordinary sensor smoothing; it is the recovery of a latent trajectory that can be meaningfully expressed in the visual coordinate system despite the absence of direct visual observations of the target (Zhang et al., 18 Sep 2025).

In the extended OST formulation, out-of-sight trajectory denoising is defined as refining the noisy positioning trajectory $S_{A_o}^{t_s:t_e}$ of an out-of-sight agent $A_o$ into a denoised latent trajectory $\hat{S}_{A_o}^{t_s:t_e}$, then projecting it into the visual coordinate system so that a downstream predictor can forecast future trajectories in image space (Zhang et al., 18 Sep 2025). The earlier OOSTraj paper presents the same logic with notation $P_o$, $S_{P_o}^{t_s:t_e}$, and $\hat{S}_{P_o}^{t_s:t_e}$ (Zhang et al., 2024).

This setting is motivated by failure modes of conventional trajectory prediction pipelines. Existing methods often assume precise, complete, and relatively clean observations, whereas the OST problem assumes noisy sensor trajectories, missing visual observations for the target agent, and no ground-truth denoised sensor labels (Zhang et al., 18 Sep 2025). The paper explicitly emphasizes that GPS can have meter-level errors and odometers accumulate drift, making the problem safety-critical in autonomous driving, surveillance, robotics, and virtual reality (Zhang et al., 18 Sep 2025).

A frequent misconception is that the module is a generic denoiser that merely cleans sensor trajectories before prediction. The papers describe something more specific. VPD is the mechanism that uses visible agents as cross-modal references to infer a scene-dependent mapping between positioning space and image space, then uses that mapping as indirect supervision for denoising the target out-of-sight trajectory (Zhang et al., 2024). This means the denoising stage is inseparable from cross-modal geometry.

2. Geometric formulation and vision-positioning mapping

The geometric core of VPD is a mapping from positioning coordinates, treated as 3D world coordinates, into 2D image coordinates. The papers define several coordinate systems explicitly. Noisy localization trajectories $S_{A_n}^{t_s:t_e}$ and denoised trajectories $\hat{S}_{A_n}^{t_s:t_e}$ are treated as 3D points $P \in \mathbb{R}^{3 \times 1}$, while visual trajectories $V_{A_i}^{t_s:t_e}$ are represented as 2D image points $A_o$0 (Zhang et al., 18 Sep 2025).

The basic projection equation is given as

$A_o$1

where $A_o$2 is the intrinsic matrix, $A_o$3 is the extrinsic matrix, and $A_o$4 is a scale factor (Zhang et al., 18 Sep 2025). The formulation is then combined into a single camera matrix

$A_o$5

yielding

$A_o$6

For moving cameras, the method uses a time-varying camera matrix sequence,

$A_o$7

with

$A_o$8

which the papers call the Camera Matrix Embedding (Zhang et al., 2024). This allows the same formulation to cover stationary cameras and mobile ego-cameras.

Once the denoised out-of-sight sensor trajectory is obtained, it is projected into visual space as

$A_o$9

in the OST paper (Zhang et al., 18 Sep 2025), and equivalently as

$\hat{S}_{A_o}^{t_s:t_e}$0

in OOSTraj (Zhang et al., 2024). This projection is the module’s defining operation: if $\hat{S}_{A_o}^{t_s:t_e}$1 can be estimated and $\hat{S}_{A_o}^{t_s:t_e}$2 can be denoised, then a visual-domain trajectory can be synthesized for an agent that was never visually observed during the input window.

The papers stress that most datasets do not provide $\hat{S}_{A_o}^{t_s:t_e}$3 and $\hat{S}_{A_o}^{t_s:t_e}$4, and that in moving-camera settings these may vary over time (Zhang et al., 18 Sep 2025). For that reason, the mapping is not assumed known a priori. Instead, VPD learns it from paired visual and sensor trajectories of visible agents. This yields a calibration-like mapping without relying on explicit calibration files. The stated practical simplification is that “Typically, the height dimension of the 3D point remains constant to simplify the projection from 3D to 2D” (Zhang et al., 18 Sep 2025). Beyond that, the papers do not present explicit planar homography assumptions, inverse projection, or depth recovery.

3. Internal architecture and module components

Architecturally, VPD is a composite module rather than a single block. In the extended OST paper, it consists of the Sensor Denoising Encoder (SDE), the Mapping Parameters Estimator (MPE), and the Visual Positioning Projection Module (VPP) (Zhang et al., 18 Sep 2025). In the earlier OOSTraj paper, the corresponding components are named Mobile Denoising Encoder (MDE), Camera Parameters Estimator (CPE), and VPP (Zhang et al., 2024). The naming differs, but the functional structure is the same.

The denoising encoder takes the noisy out-of-sight sensor trajectory and produces a denoised sensor-space trajectory: $\hat{S}_{A_o}^{t_s:t_e}$5 in the OST formulation (Zhang et al., 18 Sep 2025), and

$\hat{S}_{A_o}^{t_s:t_e}$6

in OOSTraj (Zhang et al., 2024). Both papers describe this encoder as a Transformer model with a fully connected layer before and after the Transformer (Zhang et al., 2024). No layer counts, hidden dimensions, numbers of heads, or positional encoding details are provided in the supplied text.

The mapping estimator predicts the camera matrix sequence from visible agents that have both modalities. In OST this is

$\hat{S}_{A_o}^{t_s:t_e}$7

(Zhang et al., 18 Sep 2025). In OOSTraj the analogous expression is

$\hat{S}_{A_o}^{t_s:t_e}$8

(Zhang et al., 2024). The key architectural point is that fusion between vision and positioning occurs through camera-matrix estimation rather than through explicit contrastive objectives, adversarial alignment, or cross-attention mechanisms (Zhang et al., 2024).

The VPP then performs the actual projection from denoised sensor coordinates to the visual plane. In the figure description of the OST paper, the authors additionally state that the denoised signal embedding undergoes sensor fusion with the mapping matrix embedding in the VPP, although the excerpt does not specify the precise fusion operator (Zhang et al., 18 Sep 2025). This omission is notable: the projection equations are explicit, but the implementation details of embedding fusion are not.

The module inputs and outputs can be summarized concisely.

Component	Input	Output
SDE / MDE	Noisy out-of-sight sensor trajectory	Denoised sensor trajectory
MPE / CPE	Paired visual and sensor trajectories of in-view agents	Camera matrix embedding $\hat{S}_{A_o}^{t_s:t_e}$9
VPP	Denoised sensor trajectory and camera matrix embedding	Projected visual trajectory

This modular decomposition is important because ablations in both papers show that performance does not arise from a generic sequence model alone. The denoiser, mapping estimator, and projection module each contribute materially to denoising and prediction quality (Zhang et al., 18 Sep 2025).

4. Supervision, training protocol, and the meaning of “unsupervised”

The papers repeatedly describe VPD as an unsupervised denoising module, but they also make clear that the phrase is specific: the denoiser is unsupervised with respect to clean sensor labels, not fully self-supervised in the strict sense (Zhang et al., 18 Sep 2025). There is no ground-truth denoised sensor trajectory $P_o$0 in realistic data, so conventional supervised denoising is unavailable. Instead, supervision is constructed in the visual domain after projection.

The denoising loss in both formulations is

$P_o$1

for OST (Zhang et al., 18 Sep 2025), and

$P_o$2

for OOSTraj (Zhang et al., 2024). Here, the projected visual trajectory is compared against the target visual trajectory. The papers explicitly state that these ground-truth visual trajectories for “out-of-sight” agents are available only during training, because out-of-sight conditions are simulated by masking or hiding an otherwise visible pedestrian (Zhang et al., 2024).

The downstream prediction decoder, called the Out-of-Sight Prediction Decoder (OPD), is also transformer-based and predicts future visual trajectories from the denoised visual signal (Zhang et al., 18 Sep 2025). The prediction loss is

$P_o$3

in the OST paper (Zhang et al., 18 Sep 2025), and equivalently in OOSTraj (Zhang et al., 2024). Both papers note a likely notation inconsistency in the OPD equation: the input-output indexing appears mismatched relative to the narrative, although the intended interpretation is clear.

The end-to-end training protocol is described operationally. During training, one agent is randomly designated as out-of-sight and its visual observations are withheld from the model. All other fully visible agents are used as in-sight references for estimating the camera mapping. The denoiser refines the held-out target’s noisy sensor trajectory, the VPP projects it to visual space, the projected trajectory is compared to the withheld true visual trajectory via $P_o$4, and OPD predicts future visual trajectories with supervision from $P_o$5 (Zhang et al., 18 Sep 2025).

A second misconception concerns temporal constraints. The papers do not introduce explicit temporal smoothness penalties, temporal filtering equations, adversarial objectives, cycle consistency, pseudo-label thresholds, or latent regularization beyond the stated losses (Zhang et al., 18 Sep 2025). Temporal coherence is handled implicitly by transformer sequence modeling rather than by an explicit smoothness term. This is an important negative result: any claim that VPD includes a Kalman-style state-space loss or a temporal consistency regularizer would go beyond the supplied material.

5. Empirical performance, ablations, and benchmark role

The experimental evidence for VPD is concentrated on the Vi-Fi and JRDB datasets. Vi-Fi is described as a multimodal vision-wireless dataset with synchronized RGB-D camera and mobile-device signals such as FTM, IMU, and noisy GPS; the paper uses noisy GPS as the sensor modality and visual points as image-space trajectories (Zhang et al., 18 Sep 2025). JRDB provides robot-centric 2D/3D annotations from cameras and LiDAR; the paper uses centers of 3D bounding boxes as noisy sensor trajectories and visual points as visual trajectories (Zhang et al., 18 Sep 2025). In both datasets, the setup is to observe the first 100 timestamps and predict the next 100, while using 100 timestamps of in-view paired trajectories for camera matrix estimation (Zhang et al., 18 Sep 2025).

The core quantitative metrics are MSE-D for denoising error on projected visual trajectories, MSE-P for future prediction error on visual trajectories, and SUM = MSE-D + MSE-P (Zhang et al., 18 Sep 2025). The extended journal version reports better results than the earlier CVPR version, and both outperform vanilla baselines.

Method	Vi-Fi (SUM / MSE-D / MSE-P)	JRDB (SUM / MSE-D / MSE-P)
Transformer	28.33 / 14.26 / 14.08	33.37 / 16.71 / 16.66
OOSTraj / prior version	27.24 / 13.42 / 13.83	25.51 / 10.52 / 14.99
OST journal version	23.09 / 11.86 / 11.23	21.97 / 10.84 / 11.13

These results support the paper’s explicit remark that denoising quality and prediction quality are positively correlated (Zhang et al., 18 Sep 2025). The module is not only improving projected trajectory denoising but also propagating gains to downstream forecasting.

The comparison with traditional denoising is also explicit. On the reported setup, Kalman Filter + Transformer achieves SUM $P_o$6, MSE-D $P_o$7, and MSE-P $P_o$8, whereas VPD + Transformer (Ours) achieves SUM $P_o$9, MSE-D $S_{P_o}^{t_s:t_e}$0, and MSE-P $S_{P_o}^{t_s:t_e}$1 (Zhang et al., 18 Sep 2025). The stated interpretation is that classical filters rely on state-transition assumptions and cannot exploit cross-modal geometric consistency from visual trajectories.

Ablations isolate the role of VPD components. In the journal version on Vi-Fi, removing SDE yields SUM $S_{P_o}^{t_s:t_e}$2, removing MPE yields $S_{P_o}^{t_s:t_e}$3, removing VPP yields $S_{P_o}^{t_s:t_e}$4, and removing OPD yields $S_{P_o}^{t_s:t_e}$5, compared with $S_{P_o}^{t_s:t_e}$6 for the full method (Zhang et al., 18 Sep 2025). On JRDB, removing MPE is especially damaging, yielding SUM $S_{P_o}^{t_s:t_e}$7 versus $S_{P_o}^{t_s:t_e}$8 for the full method (Zhang et al., 18 Sep 2025). The authors interpret this as consistent with mobile-camera settings where estimating $S_{P_o}^{t_s:t_e}$9 matters greatly.

The plug-and-play experiments are also significant. On Vi-Fi, Vanilla LSTM + 2 Stage yields SUM $\hat{S}_{P_o}^{t_s:t_e}$0, MSE-D $\hat{S}_{P_o}^{t_s:t_e}$1, MSE-P $\hat{S}_{P_o}^{t_s:t_e}$2, while Vanilla LSTM + VPD yields SUM $\hat{S}_{P_o}^{t_s:t_e}$3, MSE-D $\hat{S}_{P_o}^{t_s:t_e}$4, MSE-P $\hat{S}_{P_o}^{t_s:t_e}$5 (Zhang et al., 18 Sep 2025). On JRDB, Vanilla Transformer + 2 Stage yields SUM $\hat{S}_{P_o}^{t_s:t_e}$6, MSE-D $\hat{S}_{P_o}^{t_s:t_e}$7, MSE-P $\hat{S}_{P_o}^{t_s:t_e}$8, while Vanilla Transformer + VPD yields SUM $\hat{S}_{P_o}^{t_s:t_e}$9, MSE-D $S_{A_n}^{t_s:t_e}$0, MSE-P $S_{A_n}^{t_s:t_e}$1 (Zhang et al., 18 Sep 2025). This supports the characterization of VPD as a transferable denoising-and-projection front end rather than a feature that only works inside the authors’ own predictor.

6. Assumptions, limitations, and conceptual significance

The VPD rests on a set of explicit assumptions. First, visual trajectories are treated as relatively precise and reliable. Second, visible agents and out-of-sight agents share the same camera intrinsics and scene projection behavior within the same environment. Third, the relationship between positioning coordinates and image coordinates can be captured by a learned camera matrix embedding $S_{A_n}^{t_s:t_e}$2. Fourth, denoising can be learned by forcing projected denoised sensor trajectories to match visual trajectories during training (Zhang et al., 18 Sep 2025). These are geometric and cross-modal assumptions rather than probabilistic ones.

Several practical limitations are stated or implied. One is implicit calibration instability: intrinsic matrices are often unavailable, zoom and autofocus can alter intrinsics, and extrinsics are difficult to obtain robustly in moving systems (Zhang et al., 18 Sep 2025). Another is dependence on in-view paired trajectories. The earlier paper explicitly notes that if visual coverage is poor or no suitable visible agents exist, camera-parameter estimation may fail or become unstable (Zhang et al., 2024). A further limitation is distance generalization: the method does not require in-sight and out-of-sight agents to be at the same distance, but the extended paper states that if out-of-sight trajectories come from ranges far beyond the training distribution, performance may drop, and very long-range signals remain untested (Zhang et al., 18 Sep 2025).

The training protocol introduces an additional caveat. Out-of-sight cases are simulated by hiding otherwise visible pedestrians during training (Zhang et al., 2024). This suggests a possible train-test discrepancy between simulated out-of-sight examples and real fully out-of-view cases. The papers do not quantify that gap directly.

The conceptual significance of VPD lies in how it resolves the “missing-reference” problem. It does not hallucinate directly in image space, and it does not require clean sensor labels. Instead, it uses shared scene geometry and cross-modal alignment from other visible agents to supervise denoising indirectly (Zhang et al., 18 Sep 2025). That is why the extended paper characterizes it as the first initiative to integrate vision-positioning projection for denoising noisy sensor trajectories of out-of-sight agents (Zhang et al., 18 Sep 2025).

A plausible implication is that VPD occupies a distinct position between classical filtering and generic multimodal fusion. Relative to Kalman-style pipelines, it replaces fixed dynamical assumptions with learned scene- and time-dependent projection geometry. Relative to standard trajectory predictors, it explicitly models the transformation from positioning space to visual space before forecasting. In that sense, VPD is best understood not as a preprocessing denoiser, but as a cross-modal geometric supervision mechanism that converts noisy localization streams into visually meaningful trajectories for agents that are never directly observed by the camera (Zhang et al., 2024).