---
title: 'AnthroTAP: Automated Human Motion Point Tracking'
url: https://www.emergentmind.com/topics/anthrotap
type: topic
---

# AnthroTAP: Automated Human Motion Point Tracking

Searching arXiv for the AnthroTAP paper and closely related point-tracking context.
Searching for "AnthroTAP point tracking human motion" on arXiv.
Searching arXiv for the exact title and related works.
AnthroTAP is an automated pipeline for generating pseudo-labeled training data for point tracking from human motion. It was introduced in “Learning to Track Any Points from Human Motion” [2507.06233] as a response to a central bottleneck in point tracking: training requires large amounts of dense trajectory supervision, but manually annotating point trajectories in videos is extremely expensive. The method uses the Skinned Multi-Person Linear (SMPL) model to fit humans in video, projects 3D mesh vertices into 2D image space to obtain pseudo-trajectories, estimates visibility by ray-casting, and filters unreliable tracks using optical-flow consistency. A point tracking model trained on the resulting data achieves state-of-the-art performance on TAP-Vid while using far less data and compute than recent self-training approaches [2507.06233].

## 1. Problem setting and conceptual basis

AnthroTAP is motivated by the observation that human videos naturally contain non-rigid deformation, articulated motion, self-occlusion, occlusion by other people, clothing motion, motion blur, lighting changes, reflections, and crowd interactions. These properties make human motion a rich supervision source for point tracking, but they also make manual annotation difficult at scale. The paper therefore positions AnthroTAP as a way to automatically generate high-quality pseudo-labels from real human video rather than relying on fully manual labels or very large self-training corpora [2507.06233].

The core intuition is geometric. Each SMPL mesh vertex corresponds to a fixed anatomical location; tracking that vertex through time yields a temporally consistent 3D trajectory; projecting the trajectory into the image plane yields a 2D point track. AnthroTAP therefore converts video Human Mesh Recovery output into dense point supervision. This design treats body-surface geometry itself as the source of correspondence, rather than attempting to hand-label arbitrary image points.

A common misconception is that a human-motion-based training source should be narrowly specialized to human points. The reported evaluation argues against that simplification: training on human motion improves performance not only on human regions but also on non-human regions, with larger gains on the latter in the reported DAVIS split. This suggests that the supervisory signal extracted from articulated human motion transfers to more general point-tracking behavior [2507.06233].

## 2. Pipeline architecture

AnthroTAP consists of four stages: human detection and SMPL fitting, 3D-to-2D trajectory projection, visibility estimation by ray-casting, and trajectory filtering by optical-flow consistency. The pipeline is summarized below.

| Stage | Operation | Purpose |
|---|---|---|
| 1 | Detect humans and fit SMPL meshes | Obtain temporally consistent 3D body geometry |
| 2 | Project mesh vertices into 2D | Create pseudo point trajectories |
| 3 | Ray-cast for visibility | Mark self-occlusion and inter-person occlusion |
| 4 | Optical-flow consistency filtering | Remove unreliable tracks and transitions |

For each video frame \(I_t\), AnthroTAP applies a pretrained video Human Mesh Recovery model, specifically TokenHMR, to estimate SMPL parameters for every detected person \(p\). The body mesh for person \(p\) at frame \(t\) is denoted \(\mathcal{M}_{p,t}\), with \(N_v\) vertices. The SMPL model uses shape parameters \(\boldsymbol{\beta}_p \in \mathbb{R}^{D_\beta}\), typically constant for a person across a sequence, and pose parameters \(\boldsymbol{\theta}_{p,t} \in \mathbb{R}^{D_\theta}\), which vary over time. Vertex locations are produced by
\[
M(\boldsymbol{\beta}_p, \boldsymbol{\theta}_{p,t}) \rightarrow \{ \mathbf{v}_{p,t,j} \in \mathbb{R}^3 \}_{j=1}^{N_v}.
\]

For each mesh vertex \(j\) of person \(p\), the 3D positions across time define a trajectory \(\{\mathbf{v}_{p,t,j}\}_{t \in \mathcal{T}_{p,j}}\). AnthroTAP projects these points into the image plane with a camera projection \(\Pi : \mathbb{R}^3 \rightarrow \mathbb{R}^2\), yielding the 2D pseudo-trajectory
\[
\mathcal{X}_{p,j} = \{ \mathbf{x}_{p,t,j} = \Pi(\mathbf{v}_{p,t,j}) \mid t \in \mathcal{T}_{p,j} \}.
\]
In effect, every mesh vertex becomes a candidate image-space point track.

The training corpus built with this pipeline uses 1.4K videos from the Let’s Dance dataset. The resulting annotated set is referred to as Anthro-LD. The paper states that all videos are non-proprietary, that the training data is fully reproducible, and that it is intended to be open-sourced [2507.06233].

## 3. Visibility modeling and pseudo-label filtering

A central technical issue is that mesh projection alone does not determine whether a projected vertex is visible. AnthroTAP addresses this first with geometric visibility estimation. For a vertex \(\mathbf{v}_{p,t,j}\), a ray is cast from camera center \(\mathbf{c}\) toward the vertex,
\[
\mathbf{r}(s) = \mathbf{c} + s(\mathbf{v}_{p,t,j} - \mathbf{c}), \qquad s \ge 0.
\]
The ray is tested against all triangular faces of all human meshes in frame \(t\), across all detected people. The visibility indicator is defined so that a point is visible when no mesh face intersects the ray before the target vertex. This captures both self-occlusion and inter-person occlusion. The implementation uses the Möller–Trumbore ray-triangle intersection algorithm [2507.06233].

The paper is explicit that ray-casting only models occlusions caused by human meshes. It does not detect occlusions from furniture, scene clutter, or arbitrary objects not represented by SMPL. That limitation motivates the second filtering stage: optical-flow consistency.

AnthroTAP uses SEA-RAFT optical flow between consecutive frames \(I_t\) and \(I_{t+1}\), computing forward and backward flow fields \(\mathbf{F}_{t \to t+1}\) and \(\mathbf{F}_{t+1 \to t}\). A sampling operator \(f(\mathbf{F}, \mathbf{x})\) bilinearly samples flow at subpixel location \(\mathbf{x}\). Forward-backward consistency is checked with
\[
c_{p,t,j} = \mathbbm{1}\!\left[
\left\|
f(\mathbf{F}_{t \to t+1}, \mathbf{x}_{p,t,j}) +
f\!\left(
\mathbf{F}_{t+1 \to t},
\mathbf{x}_{p,t,j} + f(\mathbf{F}_{t \to t+1}, \mathbf{x}_{p,t,j})
\right)
\right\|_2
< \delta_{\text{cons}}
\right].
\]
This identifies locations where the optical flow itself is trustworthy.

AnthroTAP then compares HMR-derived motion and optical-flow-derived motion. For a step from \(t\) to \(t+1\),
\[
\Delta \mathbf{x}_{p,t,j}^{\text{HMR}} = \mathbf{x}_{p,t+1,j} - \mathbf{x}_{p,t,j}, \qquad
\Delta \mathbf{x}_{p,t,j}^{\text{OF}} = f(\mathbf{F}_{t \to t+1}, \mathbf{x}_{p,t,j}).
\]
Each displacement is normalized by
\[
L_{\text{short}} =
\min\left(
\|\Delta \mathbf{x}_{p,t,j}^{\text{HMR}}\|_2,\,
\|\Delta \mathbf{x}_{p,t,j}^{\text{OF}}\|_2
\right)
+ \epsilon_{\text{norm}},
\]
and a transition is flagged as erroneous when the normalized displacements differ by more than a threshold \(\tau_{\text{dist}}\). At the trajectory level, AnthroTAP computes the fraction of erroneous, visible, flow-consistent transitions,
\[
R_{p,j} =
\frac{
\sum_{k=1}^{|\mathcal{T}_{p,j}|-1}
e_{p,t_k,j}\, v_{p,t_k,j}\, c_{p,t_k,j}
}{
\sum_{k=1}^{|\mathcal{T}_{p,j}|-1}
v_{p,t_k,j}\, c_{p,t_k,j}
+ \epsilon_{\text{ratio}}
},
\]
and rejects a trajectory if \(R_{p,j} > \tau_{\text{ratio}}\).

The stated design preference is to keep fewer, higher-precision pseudo-labels rather than many noisy ones. In the paper’s interpretation, this filtering removes mesh-fitting failures, removes segments corrupted by non-human occlusion, and removes body parts poorly modeled by SMPL, especially loose clothing, hair, and accessories [2507.06233].

## 4. Training protocol and dataset characteristics

AnthroTAP is used to supervise a point tracker by fine-tuning LocoTrack. Training mixes Anthro-LD with Kubric Panning MOVi-E, with each batch sampling from the two sources with equal probability. The reported training configuration uses resolution \(256 \times 256\), 256 tracks per batch, AdamW, learning rate \(3 \times 10^{-4}\), weight decay \(1 \times 10^{-3}\), 50K training steps, a cosine learning-rate schedule with 1000-step warmup, gradient clipping with max norm 1.0, batch size 1 per GPU, and 4 NVIDIA A6000 GPUs. Training time is reported as about 1 day [2507.06233].

The filtering hyperparameters are given explicitly:
\[
\delta_{\text{cons}} = 1.0,\quad
\tau_{\text{dist}} = 1.0,\quad
\epsilon_{\text{norm}} = 2.0,\quad
\epsilon_{\text{ratio}} = 10^{-6},\quad
\tau_{\text{ratio}} = 0.25.
\]
The paper also notes that occlusion prediction is not directly supervised for points marked as occluded, because the occlusion labels are intentionally conservative and may contain inaccuracies. Supervision therefore emphasizes point positions rather than explicit occlusion labels.

The generated dataset is characterized as especially challenging. The paper reports higher trajectory complexity and higher trajectory diversity than DriveTrack, PointOdyssey, and Kubric, and states that 82% of frames contain more than one person. The accompanying interpretation is that frequent multi-person scenes, articulated motion, and inter-person occlusion produce trajectories that are richer than those in simpler rigid-object settings [2507.06233].

## 5. Empirical results and ablation findings

The main evaluation uses TAP-Vid on DAVIS, Kinetics, and RoboTAP, with metrics AJ, \(< \delta_{avg}^x >\), and OA, under both First and Strided query protocols. At \(256 \times 256\), the AnthroTAP-trained model, Anthro-LocoTrack, reports **64.8 AJ / 77.3 / 89.1 OA** on DAVIS First, **69.0 / 81.0 / 90.8** on DAVIS Strided, **53.9 / 68.4 / 86.4** on Kinetics First, and **64.7 / 79.2 / 88.4** on RoboTAP First. At \(384 \times 512\), it reports **65.9 / 78.9 / 87.3** on DAVIS First, **71.1 / 82.9 / 90.3** on DAVIS Strided, and **54.8 / 68.6 / 85.3** on Kinetics First. The paper characterizes these results as the best or near-best positional accuracy across the benchmarks [2507.06233].

The efficiency comparison is one of the method’s central empirical claims. Against BootsTAPIR, AnthroTAP uses 1.4K videos rather than about 15 million videos, described as roughly 10,000× less data, yet surpasses BootsTAPIR on TAP-Vid. Against CoTracker3, AnthroTAP uses 1.4K videos rather than about 15,000 real videos, about 11× smaller, while outperforming CoTracker3. The compute comparison is similarly large: AnthroTAP training uses 1 day on 4 A6000 GPUs, whereas the cited training setups for CoTracker3 and BootsTAPIR use 32 A100 80GB GPUs and 256 A100 GPUs, respectively [2507.06233].

The ablations sharpen the paper’s interpretation of where performance gains come from. In a human-versus-non-human analysis on TAP-Vid-DAVIS, the reported improvements on human-only points are AJ +0.5, \(< \delta_{avg}^x >\) +0.9, and OA +0.8, whereas non-human points improve by AJ +2.0, \(< \delta_{avg}^x >\) +2.1, and OA +1.4. In a direct comparison on the same Let’s Dance videos, CoTracker3 with AnthroTAP pseudo-labels reports **65.0 / 77.3 / 90.7** on DAVIS, compared with **64.2 / 76.5 / 89.6** for the original self-training strategy and **63.8 / 76.3 / 90.2** for the baseline CoTracker3. The paper’s conclusion is that simply using real videos is not enough; the quality of supervision matters. Optical-flow rejection also contributes measurably, improving a version that uses only mesh projection and ray-cast visibility from **64.4 / 76.9 / 88.6** to **64.8 / 77.3 / 89.1** [2507.06233].

The temporal ablations indicate that 48-frame clips perform best, with 24-frame and 64-frame clips slightly worse, and that frame dilation 2 performs best, whereas too large a dilation makes adjacent frames too far apart. These findings are consistent with the method’s reliance on short-range motion consistency rather than arbitrarily sparse temporal sampling.

## 6. Limitations, scope, and terminological ambiguity

The paper enumerates several limitations. Occlusion labels are imperfect because ray-casting only models occlusions from humans. The filtering is conservative and can reject valid trajectories. SMPL does not model loose clothing, hair, accessories, or other highly deformable details. Occlusion supervision is not considered reliable enough for direct training, so the method focuses on positional supervision. The resulting pseudo-labeled dataset is designed for training rather than as perfect ground truth for evaluation, which limits its utility as a benchmark in the strictest sense [2507.06233].

Within those limits, AnthroTAP is presented as effective because it combines human motion as rich supervision, SMPL-based 3D geometry for stable correspondences, and geometric plus photometric filtering for pseudo-label refinement. A plausible implication is that its gains derive less from scale in the conventional web-video sense than from the structural bias imposed by mesh-based correspondences and conservative filtering.

The name “AnthroTAP” is also ambiguous across subfields. In work on anthropomorphism in AI assistant tools, “AnthroTAP” is not introduced as a formal acronym, but is used as an interpretive label for a prompt-based walkthrough for cataloguing anthropomorphic features in AI assistant tools; the paper itself states that it does not explicitly name a formal acronym “AnthroTAP” [2502.16345]. In current arXiv usage, however, the most explicit and technical referent of AnthroTAP is the human-motion-based pseudo-labeling pipeline for point tracking described above.

Source: https://www.emergentmind.com/topics/anthrotap