TrackAny3D: Dual 3D Tracking Paradigms

• TrackAny3D is a 3D-centric tracking paradigm that represents objects using explicit 3D state variables—such as geometry and 6DoF motion—instead of traditional 2D boxes or masks.
• One formulation leverages RGB video by reconstructing a canonical 3D mesh with differentiable rendering to enable robust long-term correspondence despite occlusions and appearance changes.
• The LiDAR-based approach transfers pretrained point-cloud transformers through parameter-efficient adapters, MoGE modules, and temporal context propagation for category-unified single-object tracking.

TrackAny3D is a name used in recent arXiv literature for two distinct 3D-centric tracking formulations. In one usage, it denotes the framework introduced as “Tracking by 3D Model Estimation of Unknown Objects in Videos,” which casts monocular object tracking as joint 3D reconstruction and per-frame 6DoF pose estimation from RGB video (Rozumnyi et al., 2023). In another, it denotes “TrackAny3D: Transferring Pretrained 3D Models for Category-unified 3D Point Cloud Tracking,” a category-agnostic 3D LiDAR single-object tracker that transfers a pretrained point-cloud transformer through parameter-efficient adaptation, geometry-routed experts, and temporal context propagation (Wang et al., 26 Jul 2025). Despite differing modalities and task definitions, both usages replace weak 2D or category-specific state descriptions with richer explicit 3D state variables.

1. Terminological scope and problem formulations

In the cited literature, “TrackAny3D” does not identify a single method family with a uniform implementation. It instead labels two systems that share a common premise: tracking should be expressed in terms of 3D structure, geometry, or motion rather than only framewise 2D localization.

Usage	Input modality and task	Core tracked state
“Tracking by 3D Model Estimation of Unknown Objects in Videos” (Rozumnyi et al., 2023)	RGB video object tracking with first-frame box or mask	Canonical textured 3D mesh, texture map, and per-frame 6DoF pose
“TrackAny3D: Transferring Pretrained 3D Models for Category-unified 3D Point Cloud Tracking” (Wang et al., 26 Jul 2025)	3D LiDAR single-object tracking	Template/search point clouds and current 3D bounding-box pose
Related benchmark context: “TAPVid-3D: A Benchmark for Tracking Any Point in 3D” (Koppula et al., 2024)	Long-range query-based 3D point tracking	Full 3D trajectory of a material point plus visibility

This dual usage is not merely nominal. The RGB formulation treats tracking as inverse graphics over a canonical object model, whereas the LiDAR formulation treats tracking as category-unified geometric matching and localization in sparse point clouds. A plausible implication is that “TrackAny3D” has become shorthand for a broader research move from 2D propagation toward explicit 3D state estimation.

2. RGB-video TrackAny3D as tracking by reconstruction

The 2023 TrackAny3D formulation argues that standard model-free tracking is limited when it represents the target only by a 2D box or mask. In that framework, the tracked state is a single object shape, a single appearance texture, and a 6DoF pose sequence, so the problem becomes a dense long-term correspondence problem over all 3D surface points, including points that are temporarily invisible (Rozumnyi et al., 2023).

The representation has three latent components. First, the object shape is a single canonical 3D mesh with fixed topology and faces, deformed by vertex offsets $P$. The method starts from a sphere-like mesh with 1212 vertices and optimizes those offsets. Second, appearance is encoded as a texture map $T$. The feature extractor $F(\cdot)$ can be the identity, in which case the texture is RGB, or it can be a learned feature space; the paper reports that S2DNet features serve as a “deep surface texture” that is more robust than raw RGB under illumination changes, shadows, and viewpoint-dependent appearance variation. Third, the per-frame pose is

$N_n = (T_n, Q_n),$

where $T_n$ is 3D translation and $Q_n$ is a normalized quaternion.

The pipeline is online and causal. Given video frames $\{I_1,\dots,I_N\}$ and a first-frame annotation, it first applies a strong 2D tracker, with OSTrack + AlphaRefine as the default initialization, to obtain coarse masks $\{M_1,\dots,M_N\}$. It then stores a subset of past frames as keyframes $K_n$, chosen to be well aligned, to provide significantly different views, and to remain limited in number for efficiency. For each new frame, it optimizes the canonical mesh shape $P$, the texture $T$0, and the current pose $T$1 with ADAM for up to 500 iterations per frame, stopping early if the appearance loss is sufficiently small. The rendered silhouette of the estimated model at the estimated pose yields the refined segmentation for that frame.

A central conceptual point is that this is not merely 2D mask refinement. Because the same canonical mesh is used across all frames, each rendered pixel corresponds to a surface point on that mesh, which induces surface-level correspondences across time. This enables reasoning across self-occlusion and viewpoint changes in a way that purely 2D propagation cannot directly express.

3. Differentiable rendering, objective design, and empirical behavior in the RGB setting

Differentiable rendering is the mechanism that couples the latent 3D state to the observed video. Given shape $T$2, texture $T$3, and pose $T$4, the renderer $T$5 produces rendered appearance $T$6 and rendered silhouette $T$7. The method compares rendered appearance to observed image features $T$8 and rendered silhouette to the tracker mask $T$9, both for the current frame and for keyframes (Rozumnyi et al., 2023).

The total loss is a weighted sum of appearance, silhouette, motion, shape, and texture regularization: $F(\cdot)$0 with weights $F(\cdot)$1, $F(\cdot)$2, $F(\cdot)$3, $F(\cdot)$4, and $F(\cdot)$5. The appearance loss uses a Cauchy loss with scale $F(\cdot)$6. The silhouette loss combines $F(\cdot)$7 with a distance-transform term, included because IoU alone provides no gradient when the predicted and target silhouettes do not overlap. Motion regularization constrains pose change using $F(\cdot)$8 for translation and $F(\cdot)$9 for rotation. Shape smoothness is enforced by a Laplacian loss $N_n = (T_n, Q_n),$0, and texture smoothness by a total variation loss $N_n = (T_n, Q_n),$1.

Visibility handling follows directly from the mesh representation. Only the visible surface contributes to the rendering loss in a frame; occluded points are ignored when not visible but remain part of the canonical object model. The paper therefore characterizes the task as dense long-term correspondence over all 3D points, including invisible ones.

The reported experiments cover CDTB, Coin Tracking, in-hand object manipulation, and BOP TUD-L. On CDTB, the method improves segmentation for all tested initial trackers: CSR-DCF from $N_n = (T_n, Q_n),$2 to $N_n = (T_n, Q_n),$3, D3S from $N_n = (T_n, Q_n),$4 to $N_n = (T_n, Q_n),$5, and OSTrack from $N_n = (T_n, Q_n),$6 to $N_n = (T_n, Q_n),$7. On Coin Tracking, the reported sequence of results is baseline $N_n = (T_n, Q_n),$8, RGB features $N_n = (T_n, Q_n),$9, S2DNet features $T_n$0, and S2DNet plus flat prior $T_n$1. On the in-hand object manipulation dataset, it improves from OSTrack $T_n$2 to $T_n$3. On BOP TUD-L, the pose estimate reaches AR $T_n$4, notably without being given the 3D model and without training on the dataset. The ablations report that removing $T_n$5, $T_n$6, $T_n$7, or $T_n$8 hurts performance, and that RGB features can be worse than using no feature loss in some cases because raw RGB is unstable under lighting change.

The paper is explicit about its assumptions and failure modes. It assumes mostly rigid objects, a fixed underlying texture, and a single object of interest. It is not designed for strongly deformable objects. Reported limitations include bad initial segmentation, difficult motion, non-rigid deformation, extreme illumination or texture changes, and runtime of about 2 seconds per frame. If optimization fails to reach an appearance threshold within 500 iterations, the frame is declared a failure and the method falls back to the initial segmentation.

4. LiDAR TrackAny3D as category-unified pretrained transfer

The 2025 TrackAny3D addresses a different problem: 3D LiDAR-based single-object tracking with sparse and irregular point clouds, where geometric variation across categories is severe. The tracker receives a template point cloud $T_n$9 containing the target from the initial or a previous frame and a search point cloud $Q_n$0 from the current frame, and predicts the current 3D box $Q_n$1, parameterized by center $Q_n$2, yaw $Q_n$3, and size $Q_n$4. Because size is assumed constant across frames, only $Q_n$5 are predicted (Wang et al., 26 Jul 2025).

The stated motivation is that category-specific models are impractical: they require separate models for each class, generalize poorly in open-world settings, and degrade when naively unified because cars, pedestrians, cyclists, and vans exhibit conflicting geometry. TrackAny3D is presented as the first framework to transfer large-scale pretrained 3D models for category-agnostic 3D SOT.

Its overall pipeline is transformer-based. Patch embedding converts template and search point clouds into tokens, a learnable temporal token is concatenated with them, and the sequence is processed by a pretrained RECON encoder that remains frozen. The transfer is mediated by three additions: parameter-efficient adapters, a Mixture-of-Geometry-Experts (MoGE) module, and temporal context optimization with a temporal token and dynamic mask weighting. Search tokens are then passed to the localization head from MBPTrack.

The adapter design has two branches. One is a bottleneck MLP with down-projection, GeLU, and up-projection. The other is a token-wise gated scoring path with a scoring matrix and ReLU. The adapter output is

$Q_n$6

This design makes adaptation token-dependent rather than a fixed residual perturbation.

MoGE addresses persistent geometry gaps by routing tokens to expert FFNs according to latent geometric characteristics rather than category labels. For each MoGE layer, there are $Q_n$7 experts and only the top-$Q_n$8 are activated. In the final configuration, MoGE is inserted after the FFN at every even-numbered layer. The reported analysis indicates that pedestrians and cyclists activate different experts than cars and vans, with some experts specializing in rigid structures and others in non-rigid or deformable geometry.

Temporal context optimization adds a learnable temporal token $Q_n$9 that is propagated across frames via

$\{I_1,\dots,I_N\}$0

with $\{I_1,\dots,I_N\}$1 for the first frame. The method also uses dynamic mask weighting. The template mask $\{I_1,\dots,I_N\}$2 assigns $\{I_1,\dots,I_N\}$3 to target regions and $\{I_1,\dots,I_N\}$4 to background; the search mask $\{I_1,\dots,I_N\}$5 is initialized uniformly to $\{I_1,\dots,I_N\}$6. Learnable weights $\{I_1,\dots,I_N\}$7 and $\{I_1,\dots,I_N\}$8 modulate these priors and are added to the token embeddings. The reported ablations state that direct mask use is weaker, fully learnable masks are also inferior, and the proposed multiplicative weighting performs best.

The implementation details given in the paper include clip length $\{I_1,\dots,I_N\}$9, sampled points $\{M_1,\dots,M_N\}$0 and $\{M_1,\dots,M_N\}$1 using farthest point sampling, adapter bottleneck dimension $\{M_1,\dots,M_N\}$2, a frozen pretrained RECON backbone, and inference on a single NVIDIA RTX 3090. On KITTI, the method reports $\{M_1,\dots,M_N\}$3 mean in the category-unified setting, improving over MoCUT’s $\{M_1,\dots,M_N\}$4. On nuScenes, it reports $\{M_1,\dots,M_N\}$5 mean versus MoCUT’s $\{M_1,\dots,M_N\}$6. On Waymo Open Dataset, it reports $\{M_1,\dots,M_N\}$7 mean, better than all compared methods. The ablations further report that full fine-tuning is weaker than freezing the backbone and training adapters, that each major module improves performance, that all-layer MoGE hurts due to over-parameterization or overfitting, and that a 3-frame temporal propagation window is best.

5. Relation to 3D point tracking benchmarks and adjacent 3D tracking lines

The broader context for both TrackAny3D usages is the growing distinction between 2D tracking quality and genuine 3D motion understanding. “TAPVid-3D: A Benchmark for Tracking Any Point in 3D” formalizes this distinction by defining a task in which the model must output the full future 3D trajectory of a query-defined scene point and per-frame visibility, rather than only its 2D projection (Koppula et al., 2024). The benchmark reports 4569 clips total and 2828 videos total across Aria Digital Twin, DriveTrack, and Panoptic Studio, and evaluates predictions using $\{M_1,\dots,M_N\}$8, occlusion accuracy (OA), and $\{M_1,\dots,M_N\}$9. One of its central findings is that good 2D tracking does not imply good 3D motion understanding: for example, overall 2D-AJ scores for strong 2D trackers are far higher than the corresponding median-scaled 3D-AJ scores.

This benchmark perspective clarifies a common misconception. A correct 2D track can still correspond to a wrong 3D trajectory, and depth-only evaluation does not verify temporal correspondence of a material point. The RGB TrackAny3D addresses that issue through canonical-mesh correspondence and differentiable re-rendering; the LiDAR TrackAny3D addresses it through unified geometric localization in sparse 3D measurements. They therefore occupy different parts of the same broader problem landscape.

Other adjacent work illustrates how 3D tracking ideas specialize by domain. “Tracking People with 3D Representations” uses HMAR to recover a SMPL mesh and texture map for each person, then associates detections by transformer-aggregated 3D appearance, pose, and location embeddings, showing that 3D human-centric cues can outperform 2D cues on PoseTrack, MuPoTS, and AVA (Rajasegaran et al., 2021). “TRADE: Object Tracking with 3D Trajectory and Ground Depth Estimates for UAVs” integrates a 3D Kalman trajectory model, monocular depth cues, ground plane segmentation, and plane fitting to stabilize target following from a UAV with a single camera (Proença et al., 2022). Taken together, these works indicate that “tracking in 3D” is not a single recipe but a family of formulations spanning inverse graphics, point-cloud transfer, human mesh recovery, and geometry-aware localization.

6. Assumptions, limitations, and research significance

Both TrackAny3D formulations are explicitly constrained, though in different ways. The RGB method assumes a single fixed 3D shape across time, constant texture, and a single object of interest; it is consequently evaluated mostly on rigid-object datasets and is not designed for strongly deformable targets (Rozumnyi et al., 2023). The LiDAR method assumes constant object size across frames, relies on transfer from a static pretrained 3D model, and notes that long temporal propagation can become burdensome, excessive MoGE insertion can overfit, and pretrained 3D models still face distribution gaps between pretraining data and driving scenes (Wang et al., 26 Jul 2025).

A second misconception is that introducing 3D necessarily eliminates ambiguity. The cited results do not support that view. In the RGB setting, failure can still arise from bad initialization, difficult motion, severe non-rigidity, extreme illumination change, or optimization failure. In the LiDAR setting, unified tracking still requires careful balancing of frozen priors, expert specialization, and temporal calibration. The TAPVid-3D results reinforce this point at benchmark scale by showing that median-scaled 3D-AJ remains low for current baselines, and that global scale consistency is a major difficulty (Koppula et al., 2024).

The significance of TrackAny3D therefore lies less in a single canonical architecture than in a shared methodological reorientation. One branch treats tracking as tracking-by-reconstruction with a canonical textured mesh and differentiable rendering. Another treats it as category-unified transfer of pretrained geometric priors into sparse 3D tracking. A plausible implication is that future systems will continue combining stronger explicit 3D state representations with task-specific mechanisms for visibility reasoning, temporal consistency, and geometric adaptation, rather than treating tracking as 2D mask or box propagation alone.