Any2Track: Robust Multi-Domain Tracking

• Any2Track is a multi-domain tracking framework that decomposes into AnyTracker for nominal motion and AnyAdapter for online dynamics adaptation.
• It integrates domain-specific methods—including reinforcement learning for humanoid control, long-tail-aware detection for visual tracking, and beat-synchronous DTW for audio alignment—to address broad variability.
• The design decouples core tracking from adaptation, enabling zero-shot sim2real transfer and significant performance gains over conventional systems under unpredictable disturbances.

Any2Track most specifically denotes a two-stage reinforcement learning framework for humanoid motion tracking under disturbances, built from AnyTracker and AnyAdapter (Zhang et al., 17 Sep 2025). In the broader literature, the same label also functions as a design idea for tracking arbitrary targets under broad variability: visual tracking-any-object in the TAO benchmark via long-tail-aware detection and category-agnostic appearance matching (Du et al., 2021), and audio mix-to-track alignment for recovering constituent tracks in DJ mixes (Kim et al., 2020). This broader usage suggests a cross-domain emphasis on large target spaces, robust representation learning, and association or adaptation mechanisms that remain effective when conventional motion or boundary assumptions fail.

1. Scope and problem formulations

In humanoid control, Any2Track is defined as “Track Any motions under Any disturbances”: a framework intended to track diverse, highly dynamic, and contact-rich motions with a single policy, while remaining stable under terrains, external forces, and physical property changes, and transferring zero-shot from simulation to a real Unitree G1 humanoid (Zhang et al., 17 Sep 2025). The central decomposition is explicit. AnyTracker learns high-fidelity action execution in a nominal environment, and AnyAdapter adds online dynamics adaptability by using history-informed latent embeddings and an adapter architecture layered on top of the frozen tracker.

In visual tracking, the corresponding problem is tracking-any-object on TAO. TAO differs from conventional MOT and SOT by combining over 1200 categories, an extremely long-tailed category distribution, long videos with annotations at 1 FPS, and open-world tracking without user initialization (Du et al., 2021). Evaluation is performed with detection-style metrics (mAP) for tracks rather than classical MOT measures such as MOTA or IDF1. This alters system design priorities: detection recall becomes foundational, and motion modeling becomes comparatively unreliable because sparse annotation and camera motion undermine frame-to-frame extrapolation.

In music information retrieval, the relevant formulation is mix-to-track subsequence alignment. Here the goal is not merely to identify whether a track appears in a DJ mix, but to align the mix to its original tracks tightly enough to recover cue-points, transition lengths, segmentation, and tempo or key changes (Kim et al., 2020). The provided material explicitly states that “Any2Track” can be thought of as a general system that, given arbitrary complex audio, recovers which reference tracks were used, where they occur, and how they were transformed.

2. Visual Any2Track principles: tracking any object in TAO

The TAO solution in "1st Place Solution to ECCV-TAO-2020: Detect and Represent Any Object for Tracking" extends classical tracking-by-detection with three components: strong, long-tail-aware detection, category-agnostic appearance features, and simple appearance-based linking with tracklet-level post association (Du et al., 2021). The pipeline is entirely offline and operates at 1 FPS, matching annotation frequency and reducing error accumulation in low-frame-rate settings.

Detection is treated as the dominant factor. Two MMDetection-based detectors are used: Model-1, a gs-htc-dconv-x101-64x4d-fpn model trained on LVIS v0.5 + COCO with Balanced Group Softmax (BAGS), and Model-2, a DetectoRS model with SENet-154 + RFP trained on LVIS v1.0 and fine-tuned with BAGS. BAGS addresses the long tail by partitioning classes into frequency-based groups and replacing full-softmax competition with group-wise softmax,

$$p(y)=\frac{\exp(z_y)}{\sum_{j \in G(y)} \exp(z_j)},$$

optionally with group-balancing weights. DetectoRS contributes Recursive Feature Pyramid (RFP) and Switchable Atrous Convolution (SAC), both aimed at stronger multi-scale detection.

The tracking stage removes dependence on geometry. For each detection $d_{t,i}$, the system extracts two embeddings,

$$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$

and concatenates them: $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$ Association follows a DeepSORT-like cosine metric, but without motion cues: $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$ Tracklets are then merged by a tracklet-level PA module if non-overlapping tracklets have similar mean appearance features.

Several implementation choices are explicitly tied to TAO’s structure. Detections are filtered to retain only the 482 LVIS categories appearing in TAO, which improves performance relative to leaving the full LVIS label space intact. ReID1 is trained on YouTube-BB, GOT-10k, and ImageNet VID, while ReID2 adds the TAO training set for domain alignment. The feature ensemble ReID1+ReID2 improves discrimination further.

Empirically, the design rejects standard high-frame-rate motion tracking assumptions. Model-1 + SORT (30 FPS) attains 14.80 mAP, whereas Model-1 + ReID1 (1 FPS) attains 24.52 mAP. The full system, Model-1-2-482 + ReID1+ReID2 + PA (1 FPS), reaches 29.27 mAP on TAO validation. Oracle linkage remains much higher—57.72 mAP for combined detectors—indicating that association and false positives still leave a large gap, but also that detector complementarity is substantial (Du et al., 2021).

3. AnyTracker: general humanoid motion tracking with a single policy

AnyTracker is the first stage of the humanoid Any2Track framework. It learns a policy

$$\pi:\mathcal{G}\times \mathcal{S}\rightarrow \mathcal{A}$$

that maps robot proprioception and a next-frame reference motion to per-joint PD target offsets (Zhang et al., 17 Sep 2025). The state includes torso angular velocity, projected gravity vector, per-joint positions, per-joint velocities, and the last-frame action. The goal includes target per-joint positions $\tilde q_{t+1}$, target per-joint velocities, and target rigid-body information in the local frame. The action is converted by a PD controller to torque commands.

A central design choice is the canonicalized action space. Rather than predicting raw target angles for all 29 DoF, the policy outputs a normalized action passed through $\tanh$,

$u_t=\tanh(\pi(a_t\mid s_t,g_t)),$

and scales it element-wise by a per-joint vector $\boldsymbol{\alpha}$. The final PD target is a residual around the reference pose,

$d_{t,i}$0

This makes the policy learn a compact residual distribution around the nominal motion instead of a heterogeneous joint-space distribution with widely different magnitudes.

Motion conditioning is built from AMASS + LAFAN1. Motions requiring unavailable scene geometry are removed following PHC, while highly dynamic and contact-rich motions are retained, unlike GMT. Rather than relying only on phase variables, AnyTracker uses direct conditioning on the next-frame target, tightly coupling policy output to reference kinematics.

Training uses PPO in MuJoCo with a reward that combines tracking fidelity, safety, and regularization. Positive terms include upper-body position, lower-body position, torso roll/pitch, body rotation, body linear and angular velocity, joint position and velocity, root linear and angular velocity, root height, and feet height. Negative terms include action rate, DoF velocity rate, torques, joint position and velocity limit violations, self-collision, and episode termination. The overall reward is

$d_{t,i}$1

A second major design element is specialist-to-generalist training. For LAFAN1, specialist policies are trained per dataset-provided motion category. For AMASS, HumanML3D “VERB” labels are embedded via CLIP and clustered using K-means into 6 subsets, with one specialist per subset. A generalist is then distilled from these specialists using DAgger. This reduces optimization difficulty relative to training a single generalist from scratch across walking, running, dancing, jumping, turning, and other motion types.

Under no disturbances, AnyTracker outperforms reimplemented OmniH2O and ExBody2. The reported numbers are SR 89.23%, MPJPE 27.96 mm, and MPJVE 6.43, compared with 75.64% / 36.12 mm / 12.24 for OmniH2O and 79.68% / 34.70 mm / 13.69 for ExBody2. Removing canonicalized action spaces or distillation degrades performance, indicating that both are essential rather than incidental (Zhang et al., 17 Sep 2025).

4. AnyAdapter: history-informed online dynamics adaptation

AnyAdapter constitutes the second stage of Any2Track. Its purpose is to add online dynamics adaptability on top of a tracker trained in a single nominal environment, without retraining or modifying the base tracker’s weights (Zhang et al., 17 Sep 2025). The stated motivation is that domain randomization alone tends to produce conservative policies that sacrifice expressiveness and tracking fidelity, and lacks a mechanism to identify actual environment dynamics online.

The module is built around a history encoder $d_{t,i}$2 and a world model $d_{t,i}$3. At time $d_{t,i}$4, the history buffer is

$d_{t,i}$5

with history window size $d_{t,i}$6. The encoder produces a dynamics embedding,

$d_{t,i}$7

and the world model predicts future states autoregressively over a prediction window $d_{t,i}$8: $d_{t,i}$9 The training loss is an $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$0 rollout loss,

$$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$1

This proxy task is intended to force the latent embedding to represent terrain, friction, mass, and other properties that determine forward dynamics.

Adaptation is injected by adapter layers $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$2, inspired by LoRA. All AnyTracker weights are frozen, and adapter weights are zero-initialized so that initial behavior matches the original tracker. For a layer $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$3, the adapted output is conceptually

$$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$4

This preserves the base motion skill while allowing layer-wise corrections conditioned on inferred dynamics.

Stage 2 training introduces explicit disturbances. Terrains include floor friction sampled from $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$5 and Perlin-noise height fields with max terrain height 0.3 m, noise scale $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$6, octaves $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$7, persistence $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$8, and lacunarity $$f^{(1)}_{t,i}=\mathrm{ReID1}(b_{t,i}), \qquad f^{(2)}_{t,i}=\mathrm{ReID2}(b_{t,i}),$$9. External forces are random torso push/pull events with intervals $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$0 and velocity magnitude range $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$1. Physical property changes include joint friction scaling $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$2, armature scaling $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$3, torso CoM change $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$4 m, torso mass change $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$5 kg, and default joint position jittering $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$6 rad.

Ablations are explicit. Without the world model, dynamics embeddings become less informative and disturbance performance degrades. Without the adapter, training a single network with history also performs worse. The paper’s interpretation is that decoupling tracking skill from adaptability is not merely architectural convenience but a functional requirement for preserving expressiveness under robustness training (Zhang et al., 17 Sep 2025).

5. Sim-to-real transfer and empirical performance of humanoid Any2Track

Any2Track is deployed on a 29-DoF Unitree G1 humanoid and achieves a successful sim2real transfer in a zero-shot manner (Zhang et al., 17 Sep 2025). The simulation is built in MuJoCo with hardware-matched joint kinematics and limits, motor and proprioceptive sensing approximations, and a nominal dynamics model used in Stage 1. The real-world policy receives the same type of state input as in simulation and outputs PD targets or residuals at the required control frequency.

Real-world evaluation uses three disturbance settings: complex terrain, external constraint, and weight carrying. Complex terrain is instantiated with wooden boards, cardboard, foam, and fabric as uneven surfaces. External constraint is implemented by connecting the robot’s back to a hoist via a fixed-length rope. Weight carrying attaches a 5 kg payload to the robot’s back. Performance is measured over five motion trajectories per setting, using MPJPE and MPJVE computed from joint sensor readings through forward kinematics.

Against vanilla PPO with domain randomization, the reported real-world improvements are large. Under no disturbance, PPO yields MPJPE 29.38 mm and MPJVE 14.26 mm, whereas Any2Track yields 17.38 mm and 11.56. Under complex terrain, the comparison is 37.21 / 15.75 versus 18.34 / 11.87. Under external constraint, it is 39.84 / 16.01 versus 19.17 / 13.04. Under weight carrying, it is 37.52 / 16.91 versus 23.24 / 12.69. The improvements grow with disturbance severity, which the paper interprets as evidence that online dynamics adaptation is effective in sim2real deployment (Zhang et al., 17 Sep 2025).

Simulation under disturbances shows the same pattern. Under no disturbance, Any2Track reports SR 89.8%, MPJPE 16.46, and MPJVE 6.04. Under terrains, it reports SR 83.2%, MPJPE 20.68, MPJVE 7.82. Under external forces, SR 59.0%, MPJPE 28.97, MPJVE 10.81. Under physical property changes, SR 80.6%, MPJPE 27.75. The paper emphasizes that Any2Track outperforms RMA Stage 1, which has access to privileged environment factors, suggesting that the learned history-based embeddings are highly informative (Zhang et al., 17 Sep 2025).

6. Audio Any2Track: mix-to-track subsequence alignment

In audio analysis, the Any2Track idea is instantiated as mix-to-track subsequence alignment. A DJ mix is aligned against its original tracks using beat-synchronous DTW, allowing fine-grained correspondences despite tempo or key changes (Kim et al., 2020). The feature pipeline uses beat tracking and tempo estimation, Chroma Energy Normalized Statistics (CENS), and MFCC with dimensionality 12. Beat-synchronous representation provides tempo robustness, lower computational cost, and smoothing against crossfades, crowd noise, and minor timbral changes.

Given a mix feature sequence $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$7 and a track feature sequence $f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$8, subsequence DTW seeks a monotone path

$f_{t,i}=[f^{(1)}_{t,i};f^{(2)}_{t,i}].$9

that minimizes

$$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$0

with the crucial boundary condition that the track may align to any contiguous subsequence of the mix. To obtain key invariance, the chroma of the reference track is circularly shifted over all 12 semitone transpositions,

$$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$1

and the alignment with the minimum cost is selected.

The paper defines a match rate from path geometry: the ratio of diagonal moves to total moves. Tracks with match rate < 0.4 are treated as non-matching or unreliable and filtered out. This criterion is central both for dataset cleaning and for cue extraction.

Cue-point extraction is built from path stability. For a transition from track $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$2 to track $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$3, cue-out is the last beat on $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$4’s path whose preceding 32 beats are all diagonal moves; cue-in is the first beat on $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$5’s path whose succeeding 32 beats are all diagonal moves; and cue-mid is their midpoint. The transition length is then cue-in minus cue-out. The 32-beat window encodes phrase-level regularity common in EDM.

The analysis is conducted at scale on 1,557 mixes, 13,728 tracks, and 20,765 transitions after filtering. Using chroma + MFCC with key-invariant, the median time differences from human-annotated boundaries are approximately 23.38 s for cue-out, 11.40 s for cue-in, 12.16 s for cue-mid, and 4.38 s for cue-best. Cue-in hit rates are approximately 0.588 within 15 s, 0.804 within 30 s, and 0.929 within 60 s. The data also show that 86.1% of tracks are tempo-adjusted by less than 5%, 94.5% by less than 10%, and 98.6% by less than 20%; only 2.5% of played tracks are transposed at all, and 94.3% of those are shifted by 1 semitone. Transition lengths peak at multiples of 32 beats, and cue-point choices by different DJs show substantial agreement, with 23.6% of cue-point pairs having zero deviation and 40.4% within 4 beats (Kim et al., 2020).

7. Cross-domain design patterns, limitations, and significance

Taken together, these works suggest a broader Any2Track pattern: tracking performance improves when the system first learns a strong representation of the target domain and only then performs linking, adaptation, or post hoc correction. In TAO, the sequence is detection $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$6 appearance representation $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$7 appearance-only association $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$8 tracklet post association (Du et al., 2021). In humanoid control, it is nominal motion tracking $$C_{ki}=1-\frac{\bar f_k^\top f_{t,i}}{\|\bar f_k\|\,\|f_{t,i}\|}.$$9 history-informed dynamics embedding $$\pi:\mathcal{G}\times \mathcal{S}\rightarrow \mathcal{A}$$0 adapter-based correction (Zhang et al., 17 Sep 2025). In audio, it is beat-synchronous representation $$\pi:\mathcal{G}\times \mathcal{S}\rightarrow \mathcal{A}$$1 subsequence alignment $$\pi:\mathcal{G}\times \mathcal{S}\rightarrow \mathcal{A}$$2 cue and transition inference (Kim et al., 2020). This suggests that Any2Track is less a single algorithm than a recurring systems principle for broad-vocabulary tracking problems.

A second shared feature is the rejection of naive motion assumptions. TAO abandons Kalman-style geometry because 1 FPS annotations, camera motion, and irregular object motion make motion extrapolation unreliable. The humanoid framework treats adaptability as a separate learned capability rather than assuming a fixed forward model or relying solely on domain randomization. The audio system aligns in beat space, not raw time, because beat-synchronous features are less sensitive to tempo change. In each case, the temporal model is chosen to match the structure of the data rather than inherited from a conventional benchmark.

The limitations are correspondingly domain-specific. The TAO method notes that absence of motion modeling can hurt when appearance changes dramatically, and that high-recall detection can propagate false positives through appearance-only association (Du et al., 2021). Humanoid Any2Track depends on disturbance ranges seen during training and does not explicitly model sensor latency or communication delays; very aggressive disturbances or scene-rich interactions remain challenging (Zhang et al., 17 Sep 2025). The audio alignment method is biased toward beat-centric genres such as House and Trance, and can degrade under heavy remixing, extended overlap, weak harmonic content, or wrong track versions (Kim et al., 2020).

The combined significance of these systems lies in showing that “tracking any” problems are tractable when vocabulary scale, long-tail structure, and disturbance robustness are addressed directly. In vision this means long-tail-aware detection and category-agnostic embeddings; in control it means decoupling expressive motion skill from online dynamics adaptation; in audio it means replacing coarse track identification with subsequence alignment rich enough to recover structure and transformations. The shared implication is that broad-coverage tracking is achievable not by a universal metric alone, but by coupling domain-appropriate representations with lightweight yet carefully targeted association or adaptation mechanisms.