ProxyPose: Tracking 3D Motion Without 3D Models
ProxyPose reframes 6-DoF pose tracking as a video translation problem, using a finetuned diffusion model to generate proxy videos of known geometric primitives that mimic the motion of user-selected surface regions. By translating arbitrary video input into controllable proxy sequences, the method sidesteps the need for CAD models, depth maps, object segmentation, or foundation models, achieving state-of-the-art tracking accuracy on challenging benchmarks while requiring only monocular RGB input and synthetic training data.Script
Tracking the 3D pose of arbitrary objects in video has always required something extra: depth sensors, CAD models, or segmentation masks. ProxyPose throws out that requirement entirely by generating a synthetic stand-in that moves exactly like the real thing.
The core insight is elegant: instead of detecting an object's pose directly, the system learns to translate your video into a proxy video where a known geometric primitive executes the same motion. Because the cube's shape and colors are fully controlled, classical geometric solvers can extract precise 6-DoF poses frame by frame.
Training relies on synthetic data pairing source videos with proxy sequences, finetuning a 14 billion parameter video diffusion model using LoRA adapters. The model learns spatiotemporal motion priors powerful enough to generalize across textureless, specular, transparent, and even deforming surfaces without ever seeing them during training.
On the HO3D and YCBInEOAT benchmarks, ProxyPose achieves state-of-the-art accuracy with absolute translation error under 16 millimeters and rotation error around 5 degrees, beating every baseline that requires depth or 3D models. It's the only method that needs just monocular RGB input and still wins.
The method generalizes far beyond clean object-centric tracking. It handles event cameras, photon-limited scenes, face pose estimation, and even camera trajectory recovery in videos where traditional structure-from-motion completely fails. Scale and real-time efficiency remain open challenges, but the motion priors encoded in video diffusion are proving surprisingly universal.
ProxyPose reveals that generative models can serve as universal motion backbones, encoding enough geometric structure to track arbitrary 3D motion without explicit objectness priors. To dive deeper into this work and create your own video explainers, visit EmergentMind.com.
