ProxyPose: Proxy Video for 6-DoF Tracking

• The paper introduces ProxyPose, which reformulates 6-DoF pose tracking as video-to-video translation by generating a proxy video with a synthetically controlled cube.
• It employs a fine-tuned video diffusion model with LoRA adapters to synthesize the proxy, transferring challenging visual cues into an analytically tractable domain.
• Extensive benchmarks demonstrate that ProxyPose achieves state-of-the-art performance on both synthetic and real-world datasets, highlighting its robustness in diverse scenarios.

Searching arXiv for "ProxyPose" and closely related papers to ground the article in current research. ProxyPose is a method for tracking the relative 6-DoF motion of a surface region from monocular video by recasting pose tracking as video-to-video translation. Given only an RGB video and a single marked pixel in the first frame, it uses a fine-tuned video diffusion model to synthesize a proxy video in which a colored cube on a black background undergoes the same local rigid-body motion as the queried surface patch; because the proxy’s geometry and appearance are known by construction, its trajectory can then be recovered with classical pose-estimation solvers (Zhang et al., 7 Jul 2026). This formulation shifts the hardest aspects of pose tracking—including textureless, transparent, reflective, deformable, and partially occluded cases—into the learned translation stage, while preserving a deterministic downstream geometry pipeline.

1. Problem formulation and scope

ProxyPose addresses 6-DoF pose tracking from monocular video under deliberately minimal input assumptions. Given a source video and a single query pixel in frame 1, the goal is to recover a per-frame rotation and translation describing how the tracked surface region moves relative to the first frame, with rotations in $\mathrm{SO}(3)$ and translations in $\mathbb{R}^3$ (Zhang et al., 7 Jul 2026). The method assumes monocular RGB video only; one marked pixel in the first frame; no CAD or 3D model of the tracked object; no depth input; no segmentation mask for the one-query version; no object identity or object category assumption; and no requirement that the whole object be rigid.

A central conceptual distinction is that ProxyPose tracks the motion of a local surface region rather than the global pose of an entire object. The cube in the proxy video is treated as rigidly attached to the surface patch around the queried pixel, so the method estimates a 6-DoF trajectory for a local region even when the underlying scene is only locally rigid. This is the basis for its pixel-level formulation and for its applicability to non-rigid or partially articulated settings such as a breaking vase, a face, or a cartoon character.

The motivation is explicitly comparative. Prior 6-DoF methods typically require stronger inputs, including some combination of a 3D model, depth, object masks, reference images, reconstruction pipelines, or task-specific learned features. ProxyPose instead uses a generative intermediate representation: rather than regressing pose directly from difficult image evidence, it translates the input video into a synthetic domain where pose recovery becomes a standard geometric estimation problem.

2. Proxy video as an intermediate representation

The proxy video is a synthetic video depicting a simple polyhedron—implemented in all experiments as a cube with differently colored faces on a black background—that undergoes the same local rigid-body motion as the queried surface region (Zhang et al., 7 Jul 2026). Because the cube has known face colors, known vertex coordinates, and no clutter or extraneous content, downstream pose estimation reduces to detecting visible faces, establishing 2D–3D correspondences, and solving PnP.

The choice of a cube is tied to both generation and geometry. The paper states that a cube robustly presents three visible quadrilateral faces when initialized appropriately, giving up to seven non-coplanar 2D–3D correspondences. Simpler polyhedra such as tetrahedra expose too few visible faces, while more complex shapes create small faces that are harder to segment. Distinct face colors make contour detection and correspondence recovery straightforward.

The initial proxy frame is not arbitrary. ProxyPose initializes the cube so that its $[1,1,1]^\top$ diagonal is aligned with the camera ray through the query pixel, its center is placed at unit depth along that ray, and its edge length is set to $(h/\focal)s_{\text{cube}}$ with $s_{\text{cube}}=0.15$, where $h$ is image height (Zhang et al., 7 Jul 2026). This gives a predictable initial appearance and tracking scale. At inference, the implementation first extracts a $512\times512$ square crop from the input video, choosing the largest crop that keeps the query close to the center.

The camera model is also explicit. ProxyPose assumes square pixels, known aspect ratio, and a focal length $\focal$ that is known or coarsely estimated, with $\intrinsics=\mathrm{diag}(\focal,\focal,1)$. If $\focal$ is not known, the paper reports using either a $\mathbb{R}^3$0 field-of-view assumption or Depth Anything 3.

3. Generative architecture and synthetic training

ProxyPose uses Wan-14B as its pretrained video diffusion backbone and adapts it with LoRA rather than full fine-tuning (Zhang et al., 7 Jul 2026). LoRA adapters are inserted into the $\mathbb{R}^3$1, $\mathbb{R}^3$2, $\mathbb{R}^3$3, and output projections of all self-attention layers; the $\mathbb{R}^3$4, $\mathbb{R}^3$5, $\mathbb{R}^3$6, and output projections of all cross-attention layers; and the two linear layers of each feed-forward network. The main setting uses LoRA rank $\mathbb{R}^3$7, scaling factor $\mathbb{R}^3$8, and no dropout. The supplementary material states that this yields about 307M trainable parameters, around 2.2% of the frozen Wan-14B backbone.

The model jointly conditions on the source video and the proxy specification. The source and proxy are VAE-encoded, patchified into spatiotemporal tokens, concatenated, and processed by a diffusion transformer with joint self-attention. To distinguish the two streams, ProxyPose modifies Wan’s 3D rotary positional embedding to a 4D RoPE with a stream-identity axis: source tokens receive stream identifier $\mathbb{R}^3$9, and proxy tokens receive stream identifier $[1,1,1]^\top$0. A constant text prompt is also provided to the cross-attention layers, specifying a perfectly geometric cube on a black background with six fixed solid colors, sharp edges, flat faces, and no texture, gradients, reflections, shadows, or extra objects.

Training is performed only on synthetic paired data. The dataset contains 35,000 paired source and proxy videos, with source videos rendered in Blender using assets from Trellis-500K and Objaverse, and matching proxy videos rendered with PyTorch3D (Zhang et al., 7 Jul 2026). Scenes contain 1–4 foreground objects with randomized objects, backgrounds, scene composition, ground planes, camera motion, and per-object rigid motion, under HDR environments and path tracing, with two motion regimes called “drop” and “fly.” The training videos are 64 frames at $[1,1,1]^\top$1, with held-out validation and test splits having no overlap in assets, backgrounds, planes, or motion trajectories.

The learning objective is a standard flow-matching objective in which only the proxy stream is corrupted and the source stream remains clean conditioning. A critical technical device is the noise schedule offset for the first proxy frame. Rather than noising first-frame proxy tokens at timestep $[1,1,1]^\top$2, the method uses a reduced timestep $[1,1,1]^\top$3 with $[1,1,1]^\top$4. The paper identifies this as essential for stabilizing proxy identity, scale, and orientation. During training, with probability $[1,1,1]^\top$5, conditioning is dropped by zeroing source latents and also noising the first proxy frame at the global timestep instead of the anchor timestep.

Fine-tuning proceeds in three stages totaling 100k iterations: 80k iterations at $[1,1,1]^\top$6 and 29 frames, then 10k iterations at $[1,1,1]^\top$7 and 29 frames, then 10k iterations at $[1,1,1]^\top$8 and 49 frames (Zhang et al., 7 Jul 2026). Training uses $[1,1,1]^\top$9 NVIDIA H100 GPUs for about 22 GPU-days total. Inference uses 50 flow-matching denoising steps and requires around 5.5 minutes per video on a single H100.

4. Deterministic pose recovery from the proxy domain

Once the proxy video has been generated, ProxyPose reduces pose estimation to a classical vision pipeline (Zhang et al., 7 Jul 2026). Each proxy frame is analyzed by segmenting face colors, extracting quadrilateral contours, refining corners to subpixel precision, matching each detected quadrilateral to a cube face and its corner ordering, and solving PnP from the resulting 2D–3D correspondences. The paper states that this stage uses only standard OpenCV primitives.

The projection model is perspective, $(h/\focal)s_{\text{cube}}$0, under the same intrinsic model $(h/\focal)s_{\text{cube}}$1. Per-frame pose is estimated with solvePnP; the supplement specifies OpenCV’s IPPE-Square solver. A constant-velocity prior is used to improve robustness in face assignment, with translational velocity computed from previous frame-to-frame translation and rotational velocity from the Lie log of relative rotation. If no valid quadrilateral is detected, the tracker falls back to constant-velocity propagation for that frame.

The initial per-frame estimates are refined by minimizing reprojection error plus temporal smoothness. The smoothness term penalizes frame-to-frame translation changes and the matrix logarithm of relative rotation on $(h/\focal)s_{\text{cube}}$2, and optimization uses Levenberg–Marquardt with $(h/\focal)s_{\text{cube}}$3 fixed (Zhang et al., 7 Jul 2026). The supplementary material gives bundle-adjustment weights $(h/\focal)s_{\text{cube}}$4 and $(h/\focal)s_{\text{cube}}$5.

The low-level implementation is unusually concrete. Color masks use Euclidean RGB distance threshold 70; contours are extracted with cv2.findContours; quadrilateral approximation uses cv2.approxPolyDP with $(h/\focal)s_{\text{cube}}$6; minimum contour area is 500 px; and corners are refined with cv2.cornerSubPix using a $(h/\focal)s_{\text{cube}}$7 window, maximum 100 iterations, and epsilon $(h/\focal)s_{\text{cube}}$8.

ProxyPose also includes a rigidity-aware multi-query extension. When multiple queried pixels are known to lie on the same rigid body, the single-query procedure is run independently for each pixel, and the resulting proxy videos and pose tracks are fused with a bundle-adjustment formulation that introduces a per-proxy depth scalar $(h/\focal)s_{\text{cube}}$9 and optimizes a shared rigid motion. This extension is optional and used only when rigidity is known.

5. Benchmarks, quantitative results, and ablations

ProxyPose is evaluated on HO3D, YCBInEOAT, a held-out synthetic benchmark of 14 sequences, and additional in-the-wild internet videos (Zhang et al., 7 Jul 2026). Poses are expressed relative to the first frame; monocular scale ambiguity is resolved by aligning depth scale at frame 1; the query point is placed randomly on the visible object surface; and for HO3D and YCBInEOAT the evaluation uses 49-frame windows maximizing rotation change and requiring at least 90% first-frame object visibility. The reported metrics are ATE, ARE, RPE-t, RPE-r, and 2D-dist; supplementary results also include ADD and ADD-S.

The main comparisons are against FoundationPose, Any6D, One2Any, Oryon, ConceptPose, BundleSDF, SpatialTrackerV2 baselines, and custom CoTracker 3 + Depth Anything 3 baselines. In the one-query setting, ProxyPose is the only compared method using monocular RGB alone, with no 3D model, depth, or object mask.

On HO3D, ProxyPose (one query) reports ATE 15.79 mm, ARE $s_{\text{cube}}=0.15$0, RPE-t 1.193 mm, RPE-r $s_{\text{cube}}=0.15$1, and 2D-dist 8.016 px (Zhang et al., 7 Jul 2026). The paper reports that this is best on ATE, ARE, RPE-t, and RPE-r among the compared methods. With multi-query fusion, ProxyPose (two queries) gives 15.42 mm, $s_{\text{cube}}=0.15$2, 1.220 mm, $s_{\text{cube}}=0.15$3, and 12.31 px, while ProxyPose (three queries) gives 18.55 mm, $s_{\text{cube}}=0.15$4, 1.309 mm, $s_{\text{cube}}=0.15$5, and 14.63 px.

On YCBInEOAT, ProxyPose (one query) reports ATE 30.07 mm, ARE $s_{\text{cube}}=0.15$6, RPE-t 4.330 mm, RPE-r $s_{\text{cube}}=0.15$7, and 2D-dist 8.496 px (Zhang et al., 7 Jul 2026). The paper states that this one-query version again wins on ATE, RPE-t, and RPE-r, while BundleSDF is somewhat better on ARE and 2D-dist. With rigidity-aware multi-query fusion, ProxyPose (two queries) gives 31.62 mm, $s_{\text{cube}}=0.15$8, 4.820 mm, $s_{\text{cube}}=0.15$9, and 7.969 px, and ProxyPose (three queries) gives 26.32 mm, $h$0, 4.435 mm, $h$1, and 7.770 px.

Supplementary object-level results report ADD / ADD-S of 82.5 / 91.7 on HO3D and 72.4 / 84.2 on YCBInEOAT for ProxyPose (one query). On the held-out synthetic benchmark, the paper explicitly states that ProxyPose achieves the best ARE and RPE-r by a large margin. It also presents qualitative demonstrations on reflective, transparent, deformable, cartoon, underwater, and heavily occluded scenes, and reports that the method extends to face tracking, camera pose estimation, and zero-shot qualitative transfer to single-photon camera footage and event camera data.

The ablations identify the anchor schedule as the most critical design choice. On held-out synthetic data, removing the anchor schedule by setting Noise Offset 0 is described as catastrophic, with ATE 507.5 mm, ARE $h$2, RPE-t 354.5 mm, RPE-r $h$3, and 2D-dist 1009 px (Zhang et al., 7 Jul 2026). The paper also ablates LoRA rank 32, 64, and 128, finding rank 64 best, and studies training-set sizes of 300, 3k, and 35k, noting that even 300 samples still give “compelling performance.”

6. Position within proxy-based pose research and limitations

Within pose-related research, the term “proxy” has been used for several distinct intermediate representations. TCPFormer introduces an implicit pose proxy for 3D human pose lifting, where trainable latent proxy tokens mediate multiple temporal correlations between frames rather than a single temporal correlation path (Liu et al., 3 Jan 2025). PoseMaster uses a 3D skeleton made of 21 standard body bones as a sparse pose-control signal for single-image 3D character generation, and characterizes this skeleton as the nearest equivalent of a pose proxy in that setting (Yan et al., 26 Jun 2025). “Pose from Action” uses motion as a proxy supervisory signal for learning pose-sensitive appearance features without pose annotations, treating temporal transformation compatibility as the training signal that induces pose awareness (Purushwalkam et al., 2016).

This suggests that ProxyPose occupies a different part of the proxy-design space. Its proxy is not an implicit latent token bank, not a skeletal conditioning signal, and not a proxy supervision objective. Instead, it is an explicit synthetic video representation whose geometry is known and solver-friendly. The proxy is designed so that the learned component handles appearance, motion, occlusion, and material complexity, while the final 6-DoF recovery remains classical and interpretable.

The paper also states several limitations. Inference is slow, at about 5.5 minutes for proxy generation on one H100 plus pose optimization overhead. The method is not real-time, and the number of frames processed in one shot is bounded by the video generator’s context length; the reported inference windows are 49 frames (Zhang et al., 7 Jul 2026). The recovered motion is relative rather than absolute metric pose, with monocular scale ambiguity remaining unless additional constraints are added. The local-rigidity assumption breaks down on phenomena such as waves, and the discussion reports drift on fluid surfaces, some textureless or reflective objects under complex motion, and blurred fast motion. Fast motion can exceed VAE encoding capacity and degrade contour detection. The method is also sensitive to intrinsics: halving $h$4 degrades HO3D ARE from $h$5 to $h$6, with translation and 2D-dist degrading more severely than rotation. Finally, ProxyPose is not purely zero-shot; it requires task-specific fine-tuning, albeit only on synthetic paired videos.

Taken together, these properties define ProxyPose as a hybrid learned-and-geometric framework. Its main contribution is the claim that 6-DoF pose tracking can be solved by translating hard visual evidence into an analytically tractable proxy domain, rather than by estimating pose directly from the original video (Zhang et al., 7 Jul 2026).