AnthroTAP: Automated Human Motion Point Tracking

• AnthroTAP is an automated pipeline that converts 3D human mesh data into dense 2D point trajectories using SMPL models and geometric visibility estimation.
• It projects mesh vertices into the image plane and refines tracks via ray-casting and optical flow consistency to filter unreliable points.
• The approach achieves state-of-the-art performance on TAP-Vid benchmarks while using significantly less data and compute than prior self-training methods.

Searching arXiv for the AnthroTAP paper and closely related point-tracking context. Searching for "AnthroTAP point tracking human motion" on arXiv. Searching arXiv for the exact title and related works. AnthroTAP is an automated pipeline for generating pseudo-labeled training data for point tracking from human motion. It was introduced in “Learning to Track Any Points from Human Motion” (Kim et al., 8 Jul 2025) as a response to a central bottleneck in point tracking: training requires large amounts of dense trajectory supervision, but manually annotating point trajectories in videos is extremely expensive. The method uses the Skinned Multi-Person Linear (SMPL) model to fit humans in video, projects 3D mesh vertices into 2D image space to obtain pseudo-trajectories, estimates visibility by ray-casting, and filters unreliable tracks using optical-flow consistency. A point tracking model trained on the resulting data achieves state-of-the-art performance on TAP-Vid while using far less data and compute than recent self-training approaches (Kim et al., 8 Jul 2025).

1. Problem setting and conceptual basis

AnthroTAP is motivated by the observation that human videos naturally contain non-rigid deformation, articulated motion, self-occlusion, occlusion by other people, clothing motion, motion blur, lighting changes, reflections, and crowd interactions. These properties make human motion a rich supervision source for point tracking, but they also make manual annotation difficult at scale. The paper therefore positions AnthroTAP as a way to automatically generate high-quality pseudo-labels from real human video rather than relying on fully manual labels or very large self-training corpora (Kim et al., 8 Jul 2025).

The core intuition is geometric. Each SMPL mesh vertex corresponds to a fixed anatomical location; tracking that vertex through time yields a temporally consistent 3D trajectory; projecting the trajectory into the image plane yields a 2D point track. AnthroTAP therefore converts video Human Mesh Recovery output into dense point supervision. This design treats body-surface geometry itself as the source of correspondence, rather than attempting to hand-label arbitrary image points.

A common misconception is that a human-motion-based training source should be narrowly specialized to human points. The reported evaluation argues against that simplification: training on human motion improves performance not only on human regions but also on non-human regions, with larger gains on the latter in the reported DAVIS split. This suggests that the supervisory signal extracted from articulated human motion transfers to more general point-tracking behavior (Kim et al., 8 Jul 2025).

2. Pipeline architecture

AnthroTAP consists of four stages: human detection and SMPL fitting, 3D-to-2D trajectory projection, visibility estimation by ray-casting, and trajectory filtering by optical-flow consistency. The pipeline is summarized below.

Stage	Operation	Purpose
1	Detect humans and fit SMPL meshes	Obtain temporally consistent 3D body geometry
2	Project mesh vertices into 2D	Create pseudo point trajectories
3	Ray-cast for visibility	Mark self-occlusion and inter-person occlusion
4	Optical-flow consistency filtering	Remove unreliable tracks and transitions

For each video frame $I_t$, AnthroTAP applies a pretrained video Human Mesh Recovery model, specifically TokenHMR, to estimate SMPL parameters for every detected person $p$. The body mesh for person $p$ at frame $t$ is denoted $\mathcal{M}_{p,t}$, with $N_v$ vertices. The SMPL model uses shape parameters $\boldsymbol{\beta}_p \in \mathbb{R}^{D_\beta}$, typically constant for a person across a sequence, and pose parameters $$\boldsymbol{\theta}_{p,t} \in \mathbb{R}^{D_\theta}$$, which vary over time. Vertex locations are produced by

$$M(\boldsymbol{\beta}_p, \boldsymbol{\theta}_{p,t}) \rightarrow \{ \mathbf{v}_{p,t,j} \in \mathbb{R}^3 \}_{j=1}^{N_v}.$$

For each mesh vertex $j$ of person $p$0, the 3D positions across time define a trajectory $p$1. AnthroTAP projects these points into the image plane with a camera projection $p$2, yielding the 2D pseudo-trajectory

$p$3

In effect, every mesh vertex becomes a candidate image-space point track.

The training corpus built with this pipeline uses 1.4K videos from the Let’s Dance dataset. The resulting annotated set is referred to as Anthro-LD. The paper states that all videos are non-proprietary, that the training data is fully reproducible, and that it is intended to be open-sourced (Kim et al., 8 Jul 2025).

3. Visibility modeling and pseudo-label filtering

A central technical issue is that mesh projection alone does not determine whether a projected vertex is visible. AnthroTAP addresses this first with geometric visibility estimation. For a vertex $p$4, a ray is cast from camera center $p$5 toward the vertex,

$p$6

The ray is tested against all triangular faces of all human meshes in frame $p$7, across all detected people. The visibility indicator is defined so that a point is visible when no mesh face intersects the ray before the target vertex. This captures both self-occlusion and inter-person occlusion. The implementation uses the Möller–Trumbore ray-triangle intersection algorithm (Kim et al., 8 Jul 2025).

The paper is explicit that ray-casting only models occlusions caused by human meshes. It does not detect occlusions from furniture, scene clutter, or arbitrary objects not represented by SMPL. That limitation motivates the second filtering stage: optical-flow consistency.

AnthroTAP uses SEA-RAFT optical flow between consecutive frames $p$8 and $p$9, computing forward and backward flow fields $p$0 and $p$1. A sampling operator $p$2 bilinearly samples flow at subpixel location $p$3. Forward-backward consistency is checked with

$p$4

This identifies locations where the optical flow itself is trustworthy.

AnthroTAP then compares HMR-derived motion and optical-flow-derived motion. For a step from $p$5 to $p$6,

$p$7

Each displacement is normalized by

$p$8

and a transition is flagged as erroneous when the normalized displacements differ by more than a threshold $p$9. At the trajectory level, AnthroTAP computes the fraction of erroneous, visible, flow-consistent transitions,

$t$0

and rejects a trajectory if $t$1.

The stated design preference is to keep fewer, higher-precision pseudo-labels rather than many noisy ones. In the paper’s interpretation, this filtering removes mesh-fitting failures, removes segments corrupted by non-human occlusion, and removes body parts poorly modeled by SMPL, especially loose clothing, hair, and accessories (Kim et al., 8 Jul 2025).

4. Training protocol and dataset characteristics

AnthroTAP is used to supervise a point tracker by fine-tuning LocoTrack. Training mixes Anthro-LD with Kubric Panning MOVi-E, with each batch sampling from the two sources with equal probability. The reported training configuration uses resolution $t$2, 256 tracks per batch, AdamW, learning rate $t$3, weight decay $t$4, 50K training steps, a cosine learning-rate schedule with 1000-step warmup, gradient clipping with max norm 1.0, batch size 1 per GPU, and 4 NVIDIA A6000 GPUs. Training time is reported as about 1 day (Kim et al., 8 Jul 2025).

The filtering hyperparameters are given explicitly: $t$5 The paper also notes that occlusion prediction is not directly supervised for points marked as occluded, because the occlusion labels are intentionally conservative and may contain inaccuracies. Supervision therefore emphasizes point positions rather than explicit occlusion labels.

The generated dataset is characterized as especially challenging. The paper reports higher trajectory complexity and higher trajectory diversity than DriveTrack, PointOdyssey, and Kubric, and states that 82% of frames contain more than one person. The accompanying interpretation is that frequent multi-person scenes, articulated motion, and inter-person occlusion produce trajectories that are richer than those in simpler rigid-object settings (Kim et al., 8 Jul 2025).

5. Empirical results and ablation findings

The main evaluation uses TAP-Vid on DAVIS, Kinetics, and RoboTAP, with metrics AJ, $t$6, and OA, under both First and Strided query protocols. At $t$7, the AnthroTAP-trained model, Anthro-LocoTrack, reports 64.8 AJ / 77.3 / 89.1 OA on DAVIS First, 69.0 / 81.0 / 90.8 on DAVIS Strided, 53.9 / 68.4 / 86.4 on Kinetics First, and 64.7 / 79.2 / 88.4 on RoboTAP First. At $t$8, it reports 65.9 / 78.9 / 87.3 on DAVIS First, 71.1 / 82.9 / 90.3 on DAVIS Strided, and 54.8 / 68.6 / 85.3 on Kinetics First. The paper characterizes these results as the best or near-best positional accuracy across the benchmarks (Kim et al., 8 Jul 2025).

The efficiency comparison is one of the method’s central empirical claims. Against BootsTAPIR, AnthroTAP uses 1.4K videos rather than about 15 million videos, described as roughly 10,000× less data, yet surpasses BootsTAPIR on TAP-Vid. Against CoTracker3, AnthroTAP uses 1.4K videos rather than about 15,000 real videos, about 11× smaller, while outperforming CoTracker3. The compute comparison is similarly large: AnthroTAP training uses 1 day on 4 A6000 GPUs, whereas the cited training setups for CoTracker3 and BootsTAPIR use 32 A100 80GB GPUs and 256 A100 GPUs, respectively (Kim et al., 8 Jul 2025).

The ablations sharpen the paper’s interpretation of where performance gains come from. In a human-versus-non-human analysis on TAP-Vid-DAVIS, the reported improvements on human-only points are AJ +0.5, $t$9 +0.9, and OA +0.8, whereas non-human points improve by AJ +2.0, $\mathcal{M}_{p,t}$0 +2.1, and OA +1.4. In a direct comparison on the same Let’s Dance videos, CoTracker3 with AnthroTAP pseudo-labels reports 65.0 / 77.3 / 90.7 on DAVIS, compared with 64.2 / 76.5 / 89.6 for the original self-training strategy and 63.8 / 76.3 / 90.2 for the baseline CoTracker3. The paper’s conclusion is that simply using real videos is not enough; the quality of supervision matters. Optical-flow rejection also contributes measurably, improving a version that uses only mesh projection and ray-cast visibility from 64.4 / 76.9 / 88.6 to 64.8 / 77.3 / 89.1 (Kim et al., 8 Jul 2025).

The temporal ablations indicate that 48-frame clips perform best, with 24-frame and 64-frame clips slightly worse, and that frame dilation 2 performs best, whereas too large a dilation makes adjacent frames too far apart. These findings are consistent with the method’s reliance on short-range motion consistency rather than arbitrarily sparse temporal sampling.

6. Limitations, scope, and terminological ambiguity

The paper enumerates several limitations. Occlusion labels are imperfect because ray-casting only models occlusions from humans. The filtering is conservative and can reject valid trajectories. SMPL does not model loose clothing, hair, accessories, or other highly deformable details. Occlusion supervision is not considered reliable enough for direct training, so the method focuses on positional supervision. The resulting pseudo-labeled dataset is designed for training rather than as perfect ground truth for evaluation, which limits its utility as a benchmark in the strictest sense (Kim et al., 8 Jul 2025).

Within those limits, AnthroTAP is presented as effective because it combines human motion as rich supervision, SMPL-based 3D geometry for stable correspondences, and geometric plus photometric filtering for pseudo-label refinement. A plausible implication is that its gains derive less from scale in the conventional web-video sense than from the structural bias imposed by mesh-based correspondences and conservative filtering.

The name “AnthroTAP” is also ambiguous across subfields. In work on anthropomorphism in AI assistant tools, “AnthroTAP” is not introduced as a formal acronym, but is used as an interpretive label for a prompt-based walkthrough for cataloguing anthropomorphic features in AI assistant tools; the paper itself states that it does not explicitly name a formal acronym “AnthroTAP” (Maeda, 22 Feb 2025). In current arXiv usage, however, the most explicit and technical referent of AnthroTAP is the human-motion-based pseudo-labeling pipeline for point tracking described above.