Simultaneous Tracking and Predicting (STAP)

• STAP is a framework that integrates real-time tracking and prediction within a single operational loop, enhancing decision making across various domains.
• It employs techniques like predictor-assisted association and multi-hypothesis propagation to mitigate errors and improve key performance metrics.
• By incorporating direct feedback loops, STAP effectively reduces identity switches, track fragmentation, and feature outages in dynamic environments.

Searching arXiv for the main STAP papers to ground the article in the cited literature. to=arxiv_search code {"^^^^^^^^2query2 OR title:\2^^^^^^^^"Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation\"","max_results":5} {"^^^^^^^^2query2 Simultaneous Tracking And Predicting (STAP) denotes a class of formulations in which current-state estimation and short- or long-horizon forecasting are coupled within a single operational loop rather than treated as strictly separate modules. In computer vision, robotics, multi-object filtering, Wi-Fi sensing, and communication-guided localization, STAP typically means that predictions are used immediately to guide association, localization, hypothesis management, or control, while updated tracks are in turn used to refresh future predictions. The term is also acronymically ambiguous: several radar papers use “STAP” to denote Space-Time Adaptive Processing, and some of those works explicitly distinguish that usage from Simultaneous Tracking And Predicting (&&&^^^^2query2^^^^&&&, Esmaeilbeig et al., 2023, Greenewald et al., 2016).

^^^^2id:(Weng et al., 2021) OR title:\2^^^^. Terminological scope and recurrent meanings

The contemporary literature uses the same acronym for two different research lineages. In tracking-oriented work, STAP refers to simultaneous tracking and prediction, as in visual object tracking with a trajectory predictor and an assessor, multi-hypothesis trajectory prediction in autonomy stacks, ConvLSTM-based multi-target filtering, or Wi-Fi feature completion for device-free tracking. In radar, STAP usually refers instead to adaptive space-time filtering for clutter suppression; this is the dominant meaning in classical airborne radar, SAR GMTI, and dual-function radar-communication systems (Liu et al., 2020, Emambakhsh et al., 2018, Zhao et al., 8 Jul 2025, Sun et al., 2010, Liu et al., 2021).

Domain	Meaning of STAP	Representative formulation
Visual tracking	Simultaneous Tracking And Predicting	appearance tracker + trajectory predictor + assessor
Robot autonomy	Simultaneous tracking and predicting	multi-hypothesis tracking and prediction
Wi-Fi sensing	Simultaneous Tracking And Predicting	vertical/horizontal feature transfer with model-based prediction
Radar	Space-Time Adaptive Processing	joint spatial-temporal adaptive filtering

Within the tracking-oriented literature, “simultaneous” does not have a single fixed technical meaning. In some systems it denotes a unified inference objective over tracking and higher-level prediction signals, as in the joint detection-tracking-event-recognition lattice of 2^^^^2query2id:(Weng et al., 2021) OR title:\22^^^^. In others it denotes a parallel or tightly interleaved loop: an appearance tracker and a trajectory forecaster produce competing hypotheses that are resolved by an assessor; multiple tracking hypotheses are propagated into a predictor simultaneously; or a learned scene-level predictor produces the next implicit state map before a filtering update is applied (&&&^^^^2id:(Weng et al., 2021) OR title:\2query2^^^^&&&, Liu et al., 2020, &&&^^^^2query2^^^^&&&, Emambakhsh et al., 2018).

2. Core architectural patterns

A recurrent motivation for STAP is the instability of purely cascaded pipelines. In autonomous-driving prediction, using tracked past trajectories instead of ground-truth past trajectories caused an order-of-magnitude degradation on the very same targets. On KITTI IDS instances, prediction dropped from PRESERVED_PLACEHOLDER_^^^^2query2^^^^ and PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^ with ground-truth inputs to $\text{minADE}_{k}=2.820$ and $\text{minFDE}_{k}=4.514$ with tracklet inputs; on nuScenes FRAG instances, it dropped from $0.621/1.108$ to $14.520/21.815$ (&&&^^^^2query2^^^^&&&). That result gives a concrete systems-level explanation for why many later STAP formulations treat prediction as an active part of inference rather than a downstream consumer of tracking outputs.

One common pattern is predictor-assisted association. In visual tracking, the predicted current location $l_t^{pred}$ is obtained as the peak of a response map $P^t$, while the appearance tracker yields its own candidate $l_t^{app}$; an assessor then scores both candidates using the past trajectory, accumulated background motion, and the tracker heatmap, and selects the final state. In multi-person tracking by prediction, short-horizon forecasts gate detection-to-track assignment through $$d_{ij}=\min_{\tau \in \{t+1,\ldots,t+H_s\}} \|x_j-\hat p_\tau^i\|_2$$, while long-horizon predictions and context are used to merge fragmented hypotheses. In Wi-Fi sensing, a missing feature can be reconstructed from the previous value on the same link, from proportional transfer across links, or from a geometry-based predictor, and the fused feature matrix is then fed back into the tracker (Liu et al., 2020, &&&^^^^2id:(Weng et al., 2021) OR title:\2^^^^6&&&, Zhao et al., 8 Jul 2025).

A second pattern is multi-hypothesis propagation. The MTP framework replaces single-assignment tracking with multi-hypothesis data association, generates multiple full tracklet sets PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2query2^^^^ from the same affinity matrix, runs the predictor on each hypothesis, and applies a trajectory sampling step to select final outputs. The paper explicitly notes that this is not joint optimization or iterative feedback: tracking is not corrected using prediction feedback, there is no learned hypothesis weighting, and the integration is “simultaneous” because prediction is performed for all tracking hypotheses in parallel and selection considers all of them (&&&^^^^2query2^^^^&&&).

A third pattern is alternating prediction and filtering on an implicit scene representation. In the ConvLSTM-PHD formulation, the multi-target state is represented as a PHD intensity map, a ConvLSTM predicts the next probability density difference map, the resulting PHD is normalized, converted back into explicit Gaussian-mixture components, updated with measurements in a GM-PHD step, and then pruned, merged, and associated. This creates a scene-level STAP loop whose prediction unit scales with map size rather than with an explicitly enumerated target set (Emambakhsh et al., 2018).

3. Visual and video-based formulations

An early integrated formulation appeared in the joint detection-tracking-event-recognition framework of 2^^^^2query2id:(Weng et al., 2021) OR title:\22^^^^. There, tracking is defined by a dynamic-programming objective with local detection scores PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2id:(Weng et al., 2021) OR title:\2^^^^ and temporal-coherency terms PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\22^^^^, while event recognition contributes HMM emissions PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^3 and transitions PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^4. The unified objective jointly maximizes over detection indices and event states, so prediction enters in two forms: bottom-up motion prediction via PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^5 and top-down event-based prediction via HMM transitions. The paper emphasizes multidirectional information flow: detection constrains tracking, tracking regularizes detection, and event recognition biases both toward motion patterns consistent with the event (&&&^^^^2id:(Weng et al., 2021) OR title:\2query2^^^^&&&).

The 2^^^^2query22query2^^^^ visual STAP tracker made this coupling explicit for single-object tracking under occlusion and camera motion. Its system comprises a background-motion network that estimates translation, scale, and rotation, a two-stream Conv-LSTM trajectory network that predicts the current and next PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^6 locations from a past window of PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^7 frames, a pluggable appearance tracker such as DiMP or SiamMask, and an assessor that dynamically chooses between the appearance-based candidate and the trajectory-based candidate. The reported gains are concentrated in robustness: on VOT2^^^^2query2id:(Weng et al., 2021) OR title:\2^^^^9, EAO increased from PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^8 to PRESERVED_PLACEHOLDER_^^^^2id:(Weng et al., 2021) OR title:\2^^^^9 and robustness improved from $\text{minADE}_{k}=2.820$^^^^2query2^^^^ to $\text{minADE}_{k}=2.820$^^^^2id:(Weng et al., 2021) OR title:\2^^^^ when augmenting DiMP; on OTB2^^^^2query2id:(Weng et al., 2021) OR title:\2^^^^5 AUC rose from $\text{minADE}_{k}=2.820$2 to $\text{minADE}_{k}=2.820$3; on UAV^^^^2id:(Weng et al., 2021) OR title:\22^^^^3 AUC increased from $\text{minADE}_{k}=2.820$4 to $\text{minADE}_{k}=2.820$5 (Liu et al., 2020).

A different video-based formulation appears in the 2^^^^2query2id:(Weng et al., 2021) OR title:\2^^^^8 multi-person system “Tracking by Prediction.” It uses a sequential GAN to produce pedestrian probability maps and an encoder-decoder LSTM with soft attention over the target’s own history and hardwired attention over neighbors to predict short-horizon and long-horizon trajectories. Prediction is the primary cue for association rather than a post-hoc smoother: short-term forecasts gate assignments, and long-term spatial and context similarities are used to merge duplicate track hypotheses. The ablation results isolate this effect clearly: short-term prediction alone yielded MOTA $\text{minADE}_{k}=2.820$6, while adding both long-term merges raised MOTA to $\text{minADE}_{k}=2.820$7 and reduced fragmentations from $\text{minADE}_{k}=2.820$8 to $\text{minADE}_{k}=2.820$9 and identity switches from $\text{minFDE}_{k}=4.514$^^^^2query2^^^^ to $\text{minFDE}_{k}=4.514$^^^^2id:(Weng et al., 2021) OR title:\2^^^^; the full system ran at $\text{minFDE}_{k}=4.514$2 Hz on 3D MOT 2^^^^2query2id:(Weng et al., 2021) OR title:\2^^^^5 sequences (&&&^^^^2id:(Weng et al., 2021) OR title:\2^^^^6&&&).

STEP extends the STAP idea from tracking-plus-prediction to tracking-plus-pose. Starting from a known target box in the initial frame, STEP predicts target localization and anatomical keypoints in subsequent frames without per-frame detection. Its GMSP and OMRA modules remove the need for keypoint target states as input, while a transformer-based model predictor emits target-specific weights for target localization, keypoint localization, box regression, and keypoint regression. Reported pose results include OKS $\text{minFDE}_{k}=4.514$3 on APT-36K, OKS $\text{minFDE}_{k}=4.514$4 on Fish, and OKS $\text{minFDE}_{k}=4.514$5 on CrowdPose, with throughput of approximately $\text{minFDE}_{k}=4.514$6 FPS on a single RTX 4^^^^2query2^^^^9^^^^2query2^^^^ (Verma et al., 17 Mar 2025).

4. Multi-object autonomy, filtering, and SLAM

In autonomous-driving perception stacks, STAP has been studied primarily as a remedy for error propagation between multi-object tracking and trajectory prediction. The MTP framework quantifies how identity switches and fragments destabilize downstream prediction and then mitigates that instability by propagating multiple association hypotheses into the predictor. On global evaluation over all predicted objects, KITTI performance improved from $\text{minFDE}_{k}=4.514$7 to $\text{minFDE}_{k}=4.514$8 in $\text{minFDE}_{k}=4.514$9 when moving from STP $0.621/1.108$^^^^2query2^^^^ to MTP $0.621/1.108$^^^^2id:(Weng et al., 2021) OR title:\2^^^^; on nuScenes, STP at $0.621/1.108$2 improved to $0.621/1.108$3 for $0.621/1.108$4. The paper summarizes the overall nuScenes gain as “up to 34.2%,” and targeted IDS/FRAG scenarios show much larger improvements, including KITTI IDS from $0.621/1.108$5 to $0.621/1.108$6 in $0.621/1.108$7 (&&&^^^^2query2^^^^&&&).

The ConvLSTM-PHD system addresses the same simultaneous tracking/prediction problem at a different level of abstraction. It models the multi-target state as a random finite set and an implicit PHD intensity map, predicts the next probability density difference map with a ConvLSTM trained online, and then performs GM-PHD update, pruning, merging, and track management. This avoids committing to fixed per-target motion models while also avoiding per-target recurrent networks for a variable and unknown number of targets. The reported average OSPA errors are $0.621/1.108$8 on MOT^^^^2id:(Weng et al., 2021) OR title:\2^^^^5 and $0.621/1.108$9 on MOT^^^^2id:(Weng et al., 2021) OR title:\2^^^^6/^^^^2id:(Weng et al., 2021) OR title:\2^^^^7, with MOTA $14.520/21.815$^^^^2query2^^^^ on MOT^^^^2id:(Weng et al., 2021) OR title:\2^^^^6/^^^^2id:(Weng et al., 2021) OR title:\2^^^^7, $14.520/21.815$^^^^2id:(Weng et al., 2021) OR title:\2^^^^ on PNNL Parking Lot, and $14.520/21.815$2 on PETS^^^^2query2^^^^9, at approximately $14.520/21.815$3 fps on a GTX ^^^^2id:(Weng et al., 2021) OR title:\2query2^^^^8^^^^2query2^^^^ Ti (Emambakhsh et al., 2018).

GSLAMOT embeds STAP in a dynamic-scene SLAM system. It represents the environment through a semantic map for static structure, an ego trajectory from stereo VO, and an online Tracklet Graph for mobile objects. A Query Graph is built from detections, a Kalman constant-velocity model predicts active tracklets to the current frame, MSGA associates predicted tracklets and detections using neighborhood consistency, normalized generalized IoU, and ICP fitness, and OGO jointly optimizes tracklets, map, and ego trajectory over short and long windows. Reported results include Waymo Open Dataset MOTA(L^^^^2id:(Weng et al., 2021) OR title:\2^^^^) $14.520/21.815$4, MOTA(L2) $14.520/21.815$5, mismatch $14.520/21.815$6, and Traffic Congestion Dataset MOTA $14.520/21.815$7 with IDS $14.520/21.815$8, while the full parallel system averages $14.520/21.815$9 s per frame (Wang et al., 2024).

5. Wireless sensing and communication-guided tracking

In device-free Wi-Fi sensing, Baton defines STAP as a feature-completion-and-tracking loop that compensates for missing PLCR measurements across time slots and links. The vertical dimension reuses the most recent observation from the same link; the horizontal dimension exploits approximate proportionality across links between adjacent moments; and a model-based predictor computes $l_t^{pred}$^^^^2query2^^^^ from estimated position and velocity when an entire row is missing. The final prediction rule switches among observed entries, partially missing rows, and fully missing rows through reliability-weighted fusion. Implemented on Intel 53^^^^2query2query2^^^^ NICs, the system reports a median tracking error of $l_t^{pred}$^^^^2id:(Weng et al., 2021) OR title:\2^^^^ m even when the communication duty cycle is as low as $l_t^{pred}$2, and it reduces tracking error by $l_t^{pred}$3 compared with the state of the art in scenarios with severe Wi-Fi feature deficiencies (Zhao et al., 8 Jul 2025).

A closely related control-oriented formulation appears in bistatic integrated sensing and communication. There, multiple non-colocated sensing receivers estimate the vehicle’s current 2D position from AoA and delay, a maximum-likelihood fusion scheme combines selected local estimates using covariance-weighted fusion, and a constant-acceleration predictor extrapolates the next position to steer the mmWave transmit beam. The paper presents this as a simultaneous communication and tracking loop, but it also contains an explicit predict-after-estimate step that functions as a STAP mechanism for beam tracking. The proposed fusion-based scheme achieves nearly the same average spectral efficiency as an oracle trajectory-aware scheme: at $l_t^{pred}$4 dBm, Oracle is approximately $l_t^{pred}$5 bps/Hz and Fusion approximately $l_t^{pred}$6 bps/Hz; at $l_t^{pred}$7 dBm, Oracle is approximately $l_t^{pred}$8 bps/Hz and Fusion approximately $l_t^{pred}$9 bps/Hz (M et al., 2024).

These wireless formulations differ from vision-based STAP in that the tracked state is not extracted from image appearance or explicit detections. Instead, the loop is driven by feature completion, geometry-aware fusion, and kinematic extrapolation. This suggests that, outside video, STAP often denotes not only prediction-aware association but also prediction-aware sensing continuity.

6. Limits, misconceptions, and adjacent radar usage

A common misconception is that “simultaneous” necessarily implies end-to-end joint optimization. The literature does not support that interpretation. MTP explicitly states that there is no joint optimization or iterative feedback between tracking and prediction; the 2^^^^2query22query2^^^^ visual tracker chooses between appearance and prediction with a calibrated assessor rather than jointly fusing them; Baton fuses three feature estimates but still uses a separate learned tracker; and ConvLSTM-PHD alternates learned prediction with a classical GM-PHD update (&&&^^^^2query2^^^^&&&, Liu et al., 2020, Zhao et al., 8 Jul 2025, Emambakhsh et al., 2018). This suggests that STAP is better understood as a systems principle of online mutual dependence between tracking and forecasting than as a single algorithmic template.

The documented limitations are correspondingly diverse. MTP remains dependent on detection quality and on the choice of $P^t$^^^^2query2^^^^ and $P^t$^^^^2id:(Weng et al., 2021) OR title:\2^^^^, which the paper identifies as an open design problem. The visual tracker’s prediction horizon is short, and very long occlusions or abrupt motion changes reduce forecast accuracy. GSLAMOT notes that very sparse LiDAR returns or severe motion blur in VO can degrade shape and mapping residuals. Baton assumes a single dominant dynamic reflector and degrades under long structured outages or highly non-smooth motion. The fused bistatic system remains geometry-sensitive: near-degenerate geometries inflate covariance, and low SNR or multipath can invalidate local measurements (&&&^^^^2query2^^^^&&&, Liu et al., 2020, Wang et al., 2024, Zhao et al., 8 Jul 2025, M et al., 2024).

Finally, the acronym remains heavily overloaded by radar research in which STAP means Space-Time Adaptive Processing. That body of work addresses adaptive filtering, clutter suppression, waveform/filter co-design, sparse recovery of clutter spectra, Kronecker covariance modeling, and transmitter scheduling in multistatic radar networks rather than simultaneous tracking and predicting. Data-driven STAP radar treats MVDR heatmap tensors as video-like inputs and explicitly notes a natural extension to tracking and prediction, but the underlying STAP still denotes space-time adaptive processing. The same clarification is stated in cooperative CAV radar, SAR GMTI, sparse-recovery airborne radar, and DFRC waveform/filter design (Venkatasubramanian et al., 2022, Esmaeilbeig et al., 2023, Greenewald et al., 2016, Sun et al., 2010, Liu et al., 2021).

Across these literatures, the unifying idea is pragmatic rather than terminological: prediction is most useful when it is inserted directly into the tracking loop, whether by gating, reassessment, multi-hypothesis propagation, scene-level density forecasting, feature completion, or control-oriented beam steering. The papers surveyed here show that this coupling can reduce identity switches, fragments, and feature outages, but they also show that the precise meaning of “simultaneous” depends strongly on domain, representation, and inference architecture.