Adjacent Pose Tracking Method

• Adjacent pose tracking is a method that exploits temporal continuity to robustly align poses across sequential frames.
• It integrates techniques like optical flow, Bayesian filtering, and transformer-based deep learning for efficient, real-time tracking.
• Evaluations demonstrate improved accuracy, reduced ID-switches, and enhanced robustness against occlusion, motion blur, and appearance changes.

Adjacent pose tracking refers to any methodology that establishes correspondences between poses or geometric/semantic structures in sequential (typically temporally adjacent) data frames—most often video, but also event streams, depth sequences, or related spatiotemporal modalities. This class of methods exploits the temporal or spatial continuity present in real-world scenes, leveraging the predictability and smoothness of object, human, or scene dynamics to achieve more robust, computationally efficient, and accurate tracking compared to framewise or detection-only baselines. Adjacent pose tracking strategies span model-based smoothing (e.g., Kalman filtering, Bayesian recursion), optimization approaches (e.g., Lucas–Kanade, PnP solvers), particle filtering (SMC), deep learning models with explicit sequential modeling, and hybrid pipelines combining detection, local matching, and global refinement.

1. Conceptual Frameworks and Problem Formulation

The adjacent pose tracking paradigm is unified by the principle of utilizing information from one or more preceding (and occasionally succeeding) frames to inform the estimation of pose or keypoint/feature correspondences in the current frame. Generic mathematical formulations often take the form:

• Bayesian filtering: Recursive state estimation with $p(x_t|x_{t-1})$ as the motion (or process) prior and $p(I_t|x_t)$ as the measurement model (Tran et al., 2019).
• Energy minimization: Joint objective across adjacent frames,

$$E(X_t, X_{t-1}) = \sum_{i=1}^n A_i(x_i^t) + \lambda_t \sum_{i=1}^n T_i(x_i^t, x_i^{t-1}) + \lambda_s \sum_{(i,j)\in E} S_{ij}(x_i^t, x_j^t)$$

where $A_i$ is part appearance, $T_i$ is temporal displacement, and $S_{ij}$ is spatial (kinematic) dependency (Samanta et al., 2016).

Methodologies diverge in how poses are represented (keypoints, lines, 3D points, implicit structures), what features are tracked (descriptors, pixel patches, model parameters), and how adjacent-frame information is aggregated (deterministic optimization, probabilistic inference, or learned dynamics).

2. Classical Optimization and Optical Flow-based Tracking

A foundational family of adjacent pose tracking methods exploits optical flow or local image-patch matching to propagate geometric features across frames:

• Sparse Optical Flow–Based Two-Endpoint Tracking (TET): The method tracks only the endpoints of a detected line feature (from e.g. LSD), employing patch-level Lucas–Kanade optimization across Gaussian pyramids to minimize the pixel-level grayscale residual under brightness-constancy (Fu et al., 2022). The problem reduces to solving, for each endpoint,

$$E(\mathbf{m}) = \sum_{i=1}^k \big[I_2(x_i+\Delta x, y_i+\Delta y) - I_1(x_i, y_i)\big]^2$$

with iterative Gauss–Newton updates.

• Compared to descriptor-based line tracking, TET foregoes descriptor computation and per-frame re-detection, instead using a direct photometric loss at the two endpoints, resulting in a $5\times$ speed-up (13.5 ms vs. 63.8 ms per frame on standard benchmarks), with inlier rates and pose estimation accuracy within 5% of the state-of-the-art matcher (Fu et al., 2022).
• Related Lucas–Kanade and planar homography-based pose trackers are widely deployed for object and keypoint tracking in AR contexts (Ahmadyan et al., 2020).

Optical flow-guided tracking using event cameras further demonstrates high accuracy and robustness under severe motion blur and low-light conditions. This approach accumulates events into time surfaces, extracts 2D features, and propagates correspondences forward probabilistically, enabling robust 6DoF pose estimation by iterative LM minimization (Liu et al., 24 Dec 2025).

3. Model-based, Bayesian, and Learning-driven Tracking

Several approaches incorporate explicit dynamic models, probabilistic inference, and deep-sequence models:

• Bayesian Tracking with Dynamic Appearance: Projection-based face pose and animation trackers maintain a Gaussian prior–posterior update, with appearance modeled by SIFT descriptors around facial landmarks, and covariance updated via eigen-decomposition with a forgetting factor. The process enforces temporal smoothness and adaptation to gradual appearance change (Tran et al., 2019), ensuring continuity across adjacent frames and effective drift prevention.
• Sequential Monte Carlo (SMC) with Neural Predictors: Modern approaches use particle filtering, where each "track" maintains a set of diverse pose hypotheses; proposals are generated by neural sequence models (LSTM with epistemic/aleatoric uncertainty modeling), and association to pose detections leverages OKS-based likelihoods (Okada et al., 2019). On PoseTrack2018, SMC+neural proposal reduces ID-switches by 50% compared to a single-predictor baseline, confirming the practical importance of temporally constrained prediction.

Spatio-temporal pictorial structure models, explicit spatiotemporal joint relation learning, and graph-based optimizations further combine per-frame detection with pairwise spatial and temporal constraints (Sun et al., 2018, Samanta et al., 2016). The explicit modeling of joint displacements and bone vectors across frames provides additional structure and stabilizes temporal tracking.

4. Deep Learning Pipelines for Multi-instance Pose Tracking

Contemporary video pose tracking pipelines for humans and hands often rely on detection-and-tracking-by-association:

• KeyTrack operates exclusively on keypoint sequences, with a four-stage pipeline: per-frame pose estimation, parameter-free temporal keypoint refinement (OKS maximization), a compact transformer-based "pose entailment" network for online pairwise association, and a greedy track assignment procedure (Snower et al., 2019). The transformer is trained on 30-token pairs (15 keypoints per frame), with binary cross-entropy to distinguish continuation versus non-continuation; no RGB or flow is needed. This achieves SOTA accuracy (MOTA=66.6%, PoseTrack'18) at multiple orders of magnitude lower computation compared to optical-flow or full-CNN trackers.
• CondPose (for hand tracking) fuses current image crops with prior heatmaps via an attention-based branch, selecting the best prior for each current detection (brute-force scoring), and propagating the track identity via maximal confidence (Louis et al., 2021).
• Transformer-based multi-target trackers leverage per-person appearance, pose-conditioned embeddings, and attention (with learned gating between modalities) to link detections to tracks, resolving duplicate IDs and long-term occlusion (Doering et al., 2023). Evaluation demonstrates robust handling of occlusion, appearance similarity, and identity preservation (MOTA and HOTA benchmarks).

5. Adjacent-frame Tracking for 3D and 6DoF Objects

Object pose tracking in 3D is a rapidly advancing domain:

• GoTrack combines efficient frame-to-frame registration (via RAFT-style optical flow and weighted EPnP) with periodic expensive model-to-frame refinement (transformer decoder + DINOv2). Adjacent-frame tracking uses propagated correspondences and a high-inlier-ratio filter, drastically reducing compute by requiring only infrequent (2% of frames) template-to-frame registration. SOTA performance is maintained across YCB-V and HOT3D benchmarks (Nguyen et al., 8 Jun 2025).
• Reinforcement learning is employed for adjacent-point-cloud registration in TrackAgent, operating on sparse 3D data. Frame-to-frame alignment updates are discretized, with RL agents trained using PPO and reward signals computed from Chamfer distance minimization (Röhrl et al., 2023). The agent's uncertainty and mask visibility scores are deployed for automatic reinitialization triggers, delivering robust tracking and efficient automatic recovery from drift or occlusion.

3DGS-based object pose tracking (GSGTrack) leverages explicit cross-frame optimization of geometry and pose using 3D Gaussian splatting, structural graphs, and silhouette-aware losses. This architecture achieves state-of-the-art adjacent-frame robustness for monocular RGB object tracking and reconstruction, especially in challenging, depth-ambiguous scenes (Chen et al., 2024).

6. Applications, Benchmarks, and Comparative Results

Applications of adjacent pose tracking span human and hand motion analysis (PoseTrack, HO3D), autonomous driving video (KITTI, TUM, ICL-NUIM), AR/VR (e.g., mobile device instant motion tracking), and general object tracking in robotics and manipulation (YCB-V, RBOT):

Method	Speed (ms/frame)	Accuracy (MOTA/AP)	Modality	Benchmark
TET (Fu et al., 2022)	13.5	Inlier: 84%	Grayscale	TUM, KITTI, EuRoC, ICL-NUIM
KeyTrack (Snower et al., 2019)	~1000	MOTA: 66.6%, AP: 74%	Keypoints only	PoseTrack'18
GoTrack (Nguyen et al., 8 Jun 2025)	~6x faster	ADD-S: 83.6%	RGB	YCB-V, RBOT, HOT3D
TrackAgent (Röhrl et al., 2023)	0.26 s/frame	ADD-AUC: 86.8%	Depth	YCB-Video
CondPose (Louis et al., 2021)	-	mAP: 56.66%	RGB	Surgical Hands
GSGTrack (Chen et al., 2024)	-	ADD-S: 64.6%	RGB	HO3D, OnePose

Contextually, adjacent pose tracking methods consistently outperform independent single-frame estimation or basic nearest-neighbor/IoU association approaches, especially in reducing ID-switches and drift under occlusion, motion blur, or partial observability. Evaluation metrics include MOTA, AP, mean joint error (MPJPE), ADD-S/ADI (for 6DoF), rotation/translation accuracy, and qualitative continuity.

7. Robustness, Failure Modes, and Outlook

Critical to adjacent pose tracking performance is the ability to maintain robustness under circumstances such as:

• Fast motion: Exploited by sub-frame modeling and super-resolved tracking (Rozumnyi et al., 2019).
• Drift correction and occlusion: Periodic global reinitialization or model-to-frame refinement, or uncertainty-driven reset (Nguyen et al., 8 Jun 2025, Röhrl et al., 2023).
• Outliers and feature loss: Patch-level residual thresholds, RANSAC inlier ratios, and sequential attention or gating.
• Appearance change: Adaptive appearance models (covariance eigenvalue updates), dynamic template refresh, or neural uncertainty modeling.

The field continues to evolve toward architectures that tightly couple sequential structure, geometric priors, deep representation learning, and efficient association—extending beyond visible spectrum and rigid objects to deformable, articulated, or even implicit scene representations. Future research is likely to emphasize uncertainty quantification, learned dynamic priors tailored to action classes, and the fusion of multi-modal input streams for maximal adjacent-frame information utilization.