---
title: 'Any2Track: Robust Multi-Domain Tracking'
url: https://www.emergentmind.com/topics/any2track
type: topic
---

# Any2Track: Robust Multi-Domain Tracking

Any2Track most specifically denotes a two-stage reinforcement learning framework for humanoid motion tracking under disturbances, built from **AnyTracker** and **AnyAdapter** [2509.13833]. In the broader literature, the same label also functions as a design idea for tracking arbitrary targets under broad variability: visual **tracking-any-object** in the TAO benchmark via long-tail-aware detection and category-agnostic appearance matching [2101.08040], and audio **mix-to-track** alignment for recovering constituent tracks in DJ mixes [2008.10267]. This broader usage suggests a cross-domain emphasis on large target spaces, robust representation learning, and association or adaptation mechanisms that remain effective when conventional motion or boundary assumptions fail.

## 1. Scope and problem formulations

In humanoid control, Any2Track is defined as **“Track Any motions under Any disturbances”**: a framework intended to track diverse, highly dynamic, and contact-rich motions with a **single policy**, while remaining stable under terrains, external forces, and physical property changes, and transferring **zero-shot** from simulation to a real Unitree G1 humanoid [2509.13833]. The central decomposition is explicit. **AnyTracker** learns high-fidelity action execution in a nominal environment, and **AnyAdapter** adds online dynamics adaptability by using history-informed latent embeddings and an adapter architecture layered on top of the frozen tracker.

In visual tracking, the corresponding problem is **tracking-any-object** on TAO. TAO differs from conventional MOT and SOT by combining **over 1200 categories**, an extremely **long-tailed category distribution**, long videos with annotations at **1 FPS**, and **open-world tracking** without user initialization [2101.08040]. Evaluation is performed with **detection-style metrics (mAP)** for tracks rather than classical MOT measures such as MOTA or IDF1. This alters system design priorities: detection recall becomes foundational, and motion modeling becomes comparatively unreliable because sparse annotation and camera motion undermine frame-to-frame extrapolation.

In music information retrieval, the relevant formulation is **mix-to-track subsequence alignment**. Here the goal is not merely to identify whether a track appears in a DJ mix, but to align the mix to its original tracks tightly enough to recover cue-points, transition lengths, segmentation, and tempo or key changes [2008.10267]. The provided material explicitly states that **“Any2Track” can be thought of as a general system** that, given arbitrary complex audio, recovers which reference tracks were used, where they occur, and how they were transformed.

## 2. Visual Any2Track principles: tracking any object in TAO

The TAO solution in "1st Place Solution to ECCV-TAO-2020: Detect and Represent Any Object for Tracking" extends classical **tracking-by-detection** with three components: **strong, long-tail-aware detection**, **category-agnostic appearance features**, and **simple appearance-based linking with tracklet-level post association** [2101.08040]. The pipeline is entirely offline and operates at **1 FPS**, matching annotation frequency and reducing error accumulation in low-frame-rate settings.

Detection is treated as the dominant factor. Two MMDetection-based detectors are used: **Model-1**, a gs-htc-dconv-x101-64x4d-fpn model trained on **LVIS v0.5 + COCO** with **Balanced Group Softmax (BAGS)**, and **Model-2**, a **DetectoRS** model with **SENet-154 + RFP** trained on **LVIS v1.0** and fine-tuned with BAGS. BAGS addresses the long tail by partitioning classes into frequency-based groups and replacing full-softmax competition with group-wise softmax,
\[
p(y)=\frac{\exp(z_y)}{\sum_{j \in G(y)} \exp(z_j)},
\]
optionally with group-balancing weights. DetectoRS contributes **Recursive Feature Pyramid (RFP)** and **Switchable Atrous Convolution (SAC)**, both aimed at stronger multi-scale detection.

The tracking stage removes dependence on geometry. For each detection \(d_{t,i}\), the system extracts two embeddings,
\[
f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),
\]
and concatenates them:
\[
f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].
\]
Association follows a DeepSORT-like cosine metric, but **without motion cues**:
\[
C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.
\]
Tracklets are then merged by a **tracklet-level PA module** if non-overlapping tracklets have similar mean appearance features.

Several implementation choices are explicitly tied to TAO’s structure. Detections are filtered to retain only the **482 LVIS categories appearing in TAO**, which improves performance relative to leaving the full LVIS label space intact. ReID1 is trained on **YouTube-BB, GOT-10k, and ImageNet VID**, while ReID2 adds the **TAO training set** for domain alignment. The feature ensemble **ReID1+ReID2** improves discrimination further.

Empirically, the design rejects standard high-frame-rate motion tracking assumptions. **Model-1 + SORT (30 FPS)** attains **14.80 mAP**, whereas **Model-1 + ReID1 (1 FPS)** attains **24.52 mAP**. The full system, **Model-1-2-482 + ReID1+ReID2 + PA (1 FPS)**, reaches **29.27 mAP** on TAO validation. Oracle linkage remains much higher—**57.72 mAP** for combined detectors—indicating that association and false positives still leave a large gap, but also that detector complementarity is substantial [2101.08040].

## 3. AnyTracker: general humanoid motion tracking with a single policy

AnyTracker is the first stage of the humanoid Any2Track framework. It learns a policy
\[
\pi:\mathcal{G}\times \mathcal{S}\rightarrow \mathcal{A}
\]
that maps robot proprioception and a next-frame reference motion to per-joint PD target offsets [2509.13833]. The state includes torso angular velocity, projected gravity vector, per-joint positions, per-joint velocities, and the last-frame action. The goal includes target per-joint positions \(\tilde q_{t+1}\), target per-joint velocities, and target rigid-body information in the local frame. The action is converted by a PD controller to torque commands.

A central design choice is the **canonicalized action space**. Rather than predicting raw target angles for all **29 DoF**, the policy outputs a normalized action passed through \(\tanh\),
\[
u_t=\tanh(\pi(a_t\mid s_t,g_t)),
\]
and scales it element-wise by a per-joint vector \(\boldsymbol{\alpha}\). The final PD target is a residual around the reference pose,
\[
q_d=\tilde q_{t+1}+\boldsymbol{\alpha}\odot u_t.
\]
This makes the policy learn a compact residual distribution around the nominal motion instead of a heterogeneous joint-space distribution with widely different magnitudes.

Motion conditioning is built from **AMASS + LAFAN1**. Motions requiring unavailable scene geometry are removed following **PHC**, while highly dynamic and contact-rich motions are retained, unlike **GMT**. Rather than relying only on phase variables, AnyTracker uses **direct conditioning** on the next-frame target, tightly coupling policy output to reference kinematics.

Training uses **PPO** in MuJoCo with a reward that combines tracking fidelity, safety, and regularization. Positive terms include upper-body position, lower-body position, torso roll/pitch, body rotation, body linear and angular velocity, joint position and velocity, root linear and angular velocity, root height, and feet height. Negative terms include action rate, DoF velocity rate, torques, joint position and velocity limit violations, self-collision, and episode termination. The overall reward is
\[
r_t=\sum_i w_i \cdot r_{t,i}.
\]

A second major design element is **specialist-to-generalist training**. For LAFAN1, specialist policies are trained per dataset-provided motion category. For AMASS, **HumanML3D “VERB” labels** are embedded via **CLIP** and clustered using **K-means into 6 subsets**, with one specialist per subset. A generalist is then distilled from these specialists using **DAgger**. This reduces optimization difficulty relative to training a single generalist from scratch across walking, running, dancing, jumping, turning, and other motion types.

Under no disturbances, AnyTracker outperforms reimplemented **OmniH2O** and **ExBody2**. The reported numbers are **SR 89.23%**, **MPJPE 27.96 mm**, and **MPJVE 6.43**, compared with **75.64% / 36.12 mm / 12.24** for OmniH2O and **79.68% / 34.70 mm / 13.69** for ExBody2. Removing canonicalized action spaces or distillation degrades performance, indicating that both are essential rather than incidental [2509.13833].

## 4. AnyAdapter: history-informed online dynamics adaptation

AnyAdapter constitutes the second stage of Any2Track. Its purpose is to add **online dynamics adaptability** on top of a tracker trained in a single nominal environment, without retraining or modifying the base tracker’s weights [2509.13833]. The stated motivation is that domain randomization alone tends to produce **conservative policies** that sacrifice expressiveness and tracking fidelity, and lacks a mechanism to identify actual environment dynamics online.

The module is built around a **history encoder** \(\phi\) and a **world model** \(\omega\). At time \(t\), the history buffer is
\[
h_t=\{s_{t-H},a_{t-H},\ldots,s_{t-1},a_{t-1}\},
\]
with **history window size \(H=79\)**. The encoder produces a dynamics embedding,
\[
e_t=\phi(h_t),
\]
and the world model predicts future states autoregressively over a **prediction window \(N=20\)**:
\[
\hat{s}_{t+i+1}=\omega(\hat{s}_{t+i},a_{t+i},e_{t+i}), \qquad i=0,\ldots,N-1.
\]
The training loss is an \(L_1\) rollout loss,
\[
\mathcal{L}_{\mathrm{wm}}=\sum_{i=1}^{N}\|s_{t+i}-\hat{s}_{t+i}\|_1.
\]
This proxy task is intended to force the latent embedding to represent terrain, friction, mass, and other properties that determine forward dynamics.

Adaptation is injected by **adapter layers** \(\xi=\{\xi_1,\ldots,\xi_M\}\), inspired by **LoRA**. All AnyTracker weights are **frozen**, and adapter weights are **zero-initialized** so that initial behavior matches the original tracker. For a layer \(l\), the adapted output is conceptually
\[
h_l^{\mathrm{adapted}}=f_l(h_{l-1})+\xi_l(h_{l-1},e_t).
\]
This preserves the base motion skill while allowing layer-wise corrections conditioned on inferred dynamics.

Stage 2 training introduces explicit disturbances. **Terrains** include floor friction sampled from \(\mathcal{U}(0.3,2.0)\) and Perlin-noise height fields with **max terrain height 0.3 m**, noise scale \(\mathcal{U}(10.0,16.0)\), octaves \(\mathcal{U}(5.0,8.0)\), persistence \(\mathcal{U}(0.3,0.5)\), and lacunarity \(\mathcal{U}(2.0,4.0)\). **External forces** are random torso push/pull events with intervals \(\mathcal{U}(5.0,10.0)\) and velocity magnitude range \(\mathcal{U}(0.1,1.0)\). **Physical property changes** include joint friction scaling \(\mathcal{U}(0.5,2.0)\), armature scaling \(\mathcal{U}(1.0,1.05)\), torso CoM change \(\mathcal{U}(-0.15,0.15)\) m, torso mass change \(\mathcal{U}(-3.0,6.0)\) kg, and default joint position jittering \(\mathcal{U}(-0.05,0.05)\) rad.

Ablations are explicit. Without the **world model**, dynamics embeddings become less informative and disturbance performance degrades. Without the **adapter**, training a single network with history also performs worse. The paper’s interpretation is that decoupling tracking skill from adaptability is not merely architectural convenience but a functional requirement for preserving expressiveness under robustness training [2509.13833].

## 5. Sim-to-real transfer and empirical performance of humanoid Any2Track

Any2Track is deployed on a **29-DoF Unitree G1 humanoid** and achieves a **successful sim2real transfer in a zero-shot manner** [2509.13833]. The simulation is built in MuJoCo with hardware-matched joint kinematics and limits, motor and proprioceptive sensing approximations, and a nominal dynamics model used in Stage 1. The real-world policy receives the same type of state input as in simulation and outputs PD targets or residuals at the required control frequency.

Real-world evaluation uses three disturbance settings: **complex terrain**, **external constraint**, and **weight carrying**. Complex terrain is instantiated with wooden boards, cardboard, foam, and fabric as uneven surfaces. External constraint is implemented by connecting the robot’s back to a hoist via a fixed-length rope. Weight carrying attaches a **5 kg payload** to the robot’s back. Performance is measured over five motion trajectories per setting, using **MPJPE** and **MPJVE** computed from joint sensor readings through forward kinematics.

Against vanilla PPO with domain randomization, the reported real-world improvements are large. Under **no disturbance**, PPO yields **MPJPE 29.38 mm** and **MPJVE 14.26 mm**, whereas Any2Track yields **17.38 mm** and **11.56**. Under **complex terrain**, the comparison is **37.21 / 15.75** versus **18.34 / 11.87**. Under **external constraint**, it is **39.84 / 16.01** versus **19.17 / 13.04**. Under **weight carrying**, it is **37.52 / 16.91** versus **23.24 / 12.69**. The improvements grow with disturbance severity, which the paper interprets as evidence that online dynamics adaptation is effective in sim2real deployment [2509.13833].

Simulation under disturbances shows the same pattern. Under **no disturbance**, Any2Track reports **SR 89.8%**, **MPJPE 16.46**, and **MPJVE 6.04**. Under **terrains**, it reports **SR 83.2%**, **MPJPE 20.68**, **MPJVE 7.82**. Under **external forces**, **SR 59.0%**, **MPJPE 28.97**, **MPJVE 10.81**. Under **physical property changes**, **SR 80.6%**, **MPJPE 27.75**. The paper emphasizes that Any2Track outperforms **RMA Stage 1**, which has access to privileged environment factors, suggesting that the learned history-based embeddings are highly informative [2509.13833].

## 6. Audio Any2Track: mix-to-track subsequence alignment

In audio analysis, the Any2Track idea is instantiated as **mix-to-track subsequence alignment**. A DJ mix is aligned against its original tracks using **beat-synchronous DTW**, allowing fine-grained correspondences despite tempo or key changes [2008.10267]. The feature pipeline uses beat tracking and tempo estimation, **Chroma Energy Normalized Statistics (CENS)**, and **MFCC** with dimensionality **12**. Beat-synchronous representation provides tempo robustness, lower computational cost, and smoothing against crossfades, crowd noise, and minor timbral changes.

Given a mix feature sequence \(X=(x_1,\dots,x_N)\) and a track feature sequence \(Y=(y_1,\dots,y_M)\), subsequence DTW seeks a monotone path
\[
\pi=((i_1,j_1),\dots,(i_L,j_L))
\]
that minimizes
\[
C(\pi)=\sum_{l=1}^{L} c(x_{i_l},y_{j_l}),
\]
with the crucial boundary condition that the track may align to **any contiguous subsequence** of the mix. To obtain key invariance, the chroma of the reference track is circularly shifted over all **12** semitone transpositions,
\[
Y^{(k)}(t)=\mathrm{circshift}(Y^{(0)}(t),k), \qquad k\in\{0,\dots,11\},
\]
and the alignment with the minimum cost is selected.

The paper defines a **match rate** from path geometry: the ratio of diagonal moves to total moves. Tracks with **match rate < 0.4** are treated as non-matching or unreliable and filtered out. This criterion is central both for dataset cleaning and for cue extraction.

Cue-point extraction is built from path stability. For a transition from track \(A\) to track \(B\), **cue-out** is the last beat on \(A\)’s path whose preceding **32 beats** are all diagonal moves; **cue-in** is the first beat on \(B\)’s path whose succeeding **32 beats** are all diagonal moves; and **cue-mid** is their midpoint. The **transition length** is then cue-in minus cue-out. The 32-beat window encodes phrase-level regularity common in EDM.

The analysis is conducted at scale on **1,557 mixes**, **13,728 tracks**, and **20,765 transitions** after filtering. Using **chroma + MFCC with key-invariant**, the median time differences from human-annotated boundaries are approximately **23.38 s** for cue-out, **11.40 s** for cue-in, **12.16 s** for cue-mid, and **4.38 s** for cue-best. Cue-in hit rates are approximately **0.588** within **15 s**, **0.804** within **30 s**, and **0.929** within **60 s**. The data also show that **86.1%** of tracks are tempo-adjusted by less than **5%**, **94.5%** by less than **10%**, and **98.6%** by less than **20%**; only **2.5%** of played tracks are transposed at all, and **94.3%** of those are shifted by **1 semitone**. Transition lengths peak at multiples of **32 beats**, and cue-point choices by different DJs show substantial agreement, with **23.6%** of cue-point pairs having zero deviation and **40.4%** within **4 beats** [2008.10267].

## 7. Cross-domain design patterns, limitations, and significance

Taken together, these works suggest a broader Any2Track pattern: tracking performance improves when the system first learns a strong representation of the target domain and only then performs linking, adaptation, or post hoc correction. In TAO, the sequence is **detection \(\rightarrow\) appearance representation \(\rightarrow\) appearance-only association \(\rightarrow\) tracklet post association** [2101.08040]. In humanoid control, it is **nominal motion tracking \(\rightarrow\) history-informed dynamics embedding \(\rightarrow\) adapter-based correction** [2509.13833]. In audio, it is **beat-synchronous representation \(\rightarrow\) subsequence alignment \(\rightarrow\) cue and transition inference** [2008.10267]. This suggests that Any2Track is less a single algorithm than a recurring systems principle for broad-vocabulary tracking problems.

A second shared feature is the rejection of naive motion assumptions. TAO abandons Kalman-style geometry because **1 FPS** annotations, camera motion, and irregular object motion make motion extrapolation unreliable. The humanoid framework treats adaptability as a separate learned capability rather than assuming a fixed forward model or relying solely on domain randomization. The audio system aligns in **beat space**, not raw time, because beat-synchronous features are less sensitive to tempo change. In each case, the temporal model is chosen to match the structure of the data rather than inherited from a conventional benchmark.

The limitations are correspondingly domain-specific. The TAO method notes that absence of motion modeling can hurt when appearance changes dramatically, and that high-recall detection can propagate false positives through appearance-only association [2101.08040]. Humanoid Any2Track depends on disturbance ranges seen during training and does not explicitly model sensor latency or communication delays; very aggressive disturbances or scene-rich interactions remain challenging [2509.13833]. The audio alignment method is biased toward beat-centric genres such as House and Trance, and can degrade under heavy remixing, extended overlap, weak harmonic content, or wrong track versions [2008.10267].

The combined significance of these systems lies in showing that “tracking any” problems are tractable when vocabulary scale, long-tail structure, and disturbance robustness are addressed directly. In vision this means long-tail-aware detection and category-agnostic embeddings; in control it means decoupling expressive motion skill from online dynamics adaptation; in audio it means replacing coarse track identification with subsequence alignment rich enough to recover structure and transformations. The shared implication is that broad-coverage tracking is achievable not by a universal metric alone, but by coupling domain-appropriate representations with lightweight yet carefully targeted association or adaptation mechanisms.

Source: https://www.emergentmind.com/topics/any2track