PoseDiff: Unified Pose & Control Diffusion

• The paper introduces PoseDiff, a unified conditional diffusion model that maps raw RGB images to structured robot states and converts sparse video keyframes into smooth action sequences.
• It leverages a ResNet encoder, FiLM modulation, and a conditional U-Net to directly predict 3D keypoints, joint angles, and low-level control commands without auxiliary sensor inputs.
• PoseDiff achieves state-of-the-art performance on benchmarks like DREAM and Libero-Object, demonstrating real-time speeds up to 73.5 FPS and bridging perception with planning and control in robotics.

PoseDiff is a conditional diffusion model for robotics that unifies two tasks usually treated separately: robot pose estimation from a single RGB image and video-to-action inverse dynamics from sparse visual trajectories. In its pose-estimation setting, it maps raw visual observations into structured robot states such as 3D keypoints or joint angles without requiring multi-stage pipelines or auxiliary modalities. In its control setting, it conditions on sparse video keyframes generated by world models and produces smooth, continuous long-horizon action sequences through an overlap-averaging strategy, with the stated goal of bridging perception, planning, and control in embodied AI (Zhang et al., 29 Sep 2025).

1. Scope, task definition, and problem setting

PoseDiff is presented as a unified, end-to-end conditional diffusion model with two operational modes. The first is robot pose estimation, where the model predicts robot state directly from a single RGB image. The second is video-to-action inverse dynamics, where the model converts world-model-generated video keyframes into executable low-level action sequences (Zhang et al., 29 Sep 2025).

In the pose-estimation setting, the method is positioned against multi-stage or modality-dependent systems. The defining claim is that PoseDiff eliminates the need for depth maps, joint angle priors, or auxiliary sensor input, and functions efficiently in real time (Zhang et al., 29 Sep 2025). In the control setting, the method starts from sparse visual states rather than dense action supervision at inference time. The paper describes these sparse states as video keyframes generated by a world model and argues that they are too sparse to execute directly, motivating a learned inverse-dynamics module that can fill in dense trajectories (Zhang et al., 29 Sep 2025).

The experimental scope reflects this dual formulation. Robot pose estimation is evaluated on the DREAM dataset, specifically DREAM-REAL subsets including AK, XK, RS, and ORB, with ADD, AUC, and FPS as the reported metrics. Video-to-action control is evaluated on Libero-Object manipulation tasks, with conditional success rate as the reported metric (Zhang et al., 29 Sep 2025).

2. Conditional diffusion formulation and network design

PoseDiff models the robot pose as a vector $\mathbf{x}_0 \in \mathbb{R}^D$, for example a concatenation of 3D keypoints. Its forward noising process is

$$q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\left(\mathbf{x}_t; \sqrt{\bar{\alpha}_t}\mathbf{x}_0, (1-\bar{\alpha}_t)\mathbf{I}\right),$$

with $\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)$ under a prescribed noise schedule. The reverse process is defined as

$$p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t, c) = \mathcal{N}\left(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t, c), \Sigma_\theta(t)\right),$$

where $c$ denotes conditioning features extracted from the input image and $\Sigma_\theta(t) = \beta_t \mathbf{I}$ is fixed (Zhang et al., 29 Sep 2025).

Training uses the standard diffusion noise-prediction objective

$$\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{x}_0, t, \bm{\epsilon}} \left[ \| \bm{\epsilon} - \bm{\epsilon}_\theta(\mathbf{x}_t, t, c) \|_2^2 \right].$$

Architecturally, the model is composed of a ResNet visual encoder, a condition encoder that integrates visual features and diffusion timestep embeddings, FiLM modulation, and a conditional U-Net as the denoising network. FiLM is used to modulate keypoint features with scale and bias vectors derived from the conditioning pathway:

$$\mathbf{f}_\text{FiLM} = \mathbf{s} \odot \mathbf{f}_\text{kp} + \mathbf{b}.$$

The role of FiLM in the reported design is to enhance conditioning capability, while residual connections and the hierarchical encoder-decoder structure are used to improve feature integration and robustness (Zhang et al., 29 Sep 2025).

A notable aspect of the formulation is that the same denoising machinery is reused across static state estimation and action generation. This suggests a shared view of both tasks as conditional reconstruction problems over structured outputs, with visual evidence serving as the conditioning signal.

3. Pose estimation from a single RGB image

For robot pose estimation, PoseDiff takes a single RGB image and predicts structured robot state, including 3D keypoints or joint angles. The paper describes this as direct camera-to-robot state estimation, without requiring privileged information such as joint encoders at inference time (Zhang et al., 29 Sep 2025).

The evaluation protocol on DREAM-REAL uses three metrics. ADD measures millimeter error between predicted and ground-truth 3D keypoints; lower is better. AUC is the normalized area under an accuracy-threshold curve; higher is better. FPS measures throughput in images per second; higher is better (Zhang et al., 29 Sep 2025).

The reported results are summarized below.

Benchmark/metric	PoseDiff	Comparison
DREAM-REAL ADD (AK/XK/RS/ORB)	3.6 / 5.2 / 3.4 / 3.5 mm	RoboPose: 34.3 / 22.3 / 26.0 / 30.1; HoRoPose: 18.9 / 24.0 / 24.8 / 24.8; RoboPEPP: 29.0 / 22.0 / 23.0 / 27.0
DREAM-REAL AUC (AK/XK/RS/ORB)	96.4 / 94.8 / 96.6 / 96.5	RoboPEPP: 75.3 / 78.5 / 80.5 / 77.5
DREAM-REAL FPS (AK subset)	73.5 FPS	RoboPEPP: 43.5; HoRoPose: 22.6; RoboPose: 1.8

These results are reported as state of the art on DREAM, with both high accuracy and real-time speed (Zhang et al., 29 Sep 2025). The details additionally note that appendix figures show pose reconstructions that tightly match ground truth.

The significance of this mode is not only numerical. Because the model predicts structured state directly from raw RGB, it collapses perception-stage decomposition into a single denoising architecture. A plausible implication is that the method treats structured robot geometry as a diffusion target rather than as a downstream product of separately engineered detection and fitting stages.

4. Video-to-action inverse dynamics and long-horizon control

PoseDiff extends from static state estimation to inverse dynamics by taking as input pairs of sparse video keyframes generated by a world model. The details identify AVDC as the world model used in the Libero-Object experiments. Given an initial robot image and a task instruction, AVDC generates sparse video keyframes representing high-level visual states; PoseDiff then fills in the missing low-level control sequence (Zhang et al., 29 Sep 2025).

The reported mechanism is a sliding-window procedure with stride 1 and window 2. For each pair of consecutive frames, the model is modified to predict a dense sequence of $N$ low-level robot actions:

$(I_t, I_{t+1}) \rightarrow [a^1_t, \ldots, a^N_t].$

Because adjacent frame pairs share a boundary between the last action of one segment and the first action of the next, PoseDiff applies overlap-averaging to these actions before concatenating the sequence. This is described as the means of ensuring a smooth, temporally consistent trajectory (Zhang et al., 29 Sep 2025).

The Libero-Object results are reported as conditional success rates over video-valid episodes:

Task	PoseDiff	Baselines
Soup	60%	Seer Inv. Dyn.: 0%; RoboPEPP (mod): 0%
Cheese	58%	Seer Inv. Dyn.: 0%; RoboPEPP (mod): 0%
Salad	68%	Seer Inv. Dyn.: 0%; RoboPEPP (mod): 0%
Ketchup	62%	Seer Inv. Dyn.: 0%; RoboPEPP (mod): 0%
Tomato	62%	Seer Inv. Dyn.: 0%; RoboPEPP (mod): 0%

The paper’s interpretation is that other offline models accumulate error and fail entirely, whereas PoseDiff produces stable, executable long-horizon action sequences (Zhang et al., 29 Sep 2025). This is the core control-side contribution: sparse imagined visual trajectories become dense action sequences through a diffusion-based inverse-dynamics model.

5. Unified perception-control interpretation

PoseDiff is explicitly framed as a bridge between perception and control rather than as a task-specific estimator. The paper argues that pose estimation and inverse dynamics share a structural commonality: both are mappings from visual observations to structured outputs, whether those outputs are robot states or action sequences (Zhang et al., 29 Sep 2025).

This unification is operational rather than merely conceptual. The same conditional diffusion framework is used for both tasks, with changes in target representation and conditioning context rather than a change of modeling paradigm. In the state-estimation case, the target is a static robot-state vector. In the control case, the target is a dense action segment produced from a short pair of keyframes. The reported consequence is a single architecture that is scalable, accurate, and efficient across both settings (Zhang et al., 29 Sep 2025).

The paper also emphasizes real-time applicability. For pose estimation, throughput exceeds 70 FPS on the AK subset of DREAM. For control, the method is used offline to convert world-model “dreamed” videos into executable trajectories. This supports the broader embodied-AI claim that visually grounded planning modules and low-level control modules can be joined through a common structured denoising interface (Zhang et al., 29 Sep 2025).

6. Terminological distinctions and relation to similarly named diffusion methods

PoseDiff belongs to a broader family of diffusion-based pose models, but its domain and objective differ sharply from several similarly named systems. This distinction matters because the literature contains multiple “pose + diffusion” methods with different targets.

In human pose estimation, DiffPose is a conditional diffusion model for multi-hypothesis monocular 3D human pose estimation, using sampled 2D heatmap representations and an embedding transformer to model ambiguity (Holmquist et al., 2022). DRPose is a diffusion-based refinement framework for 3D human pose estimation that refines deterministic outputs and performs multi-hypothesis prediction through reverse diffusion, with SGCT and PRM as key modules (Kang et al., 2024). DPoser is a robust diffusion-based prior over SMPL pose parameters, designed for human mesh recovery, pose generation, pose completion, and motion denoising (Lu et al., 2023). SDPose fine-tunes Stable Diffusion for human pose estimation and emphasizes out-of-domain robustness through latent-space heatmap prediction and an auxiliary RGB reconstruction branch (Liang et al., 29 Sep 2025).

PoseDiff, by contrast, is not a human-pose prior or a human-pose ambiguity model. Its “pose” refers to robot state estimation and its extension to video-to-action control in embodied AI (Zhang et al., 29 Sep 2025). That distinction also separates it from adjacent diffusion-based work in person re-identification, 3D human reconstruction, and pose-conditioned generation, where pose acts as augmentation signal, reward target, or animation control rather than as robot-state output.

This difference in application domain does not make the methods unrelated. Rather, it places PoseDiff within a broader diffusion-based trend: structured pose variables are increasingly treated as denoising targets, priors, or conditioning interfaces. PoseDiff’s specific contribution is to apply that paradigm to robotics in a way that unifies state estimation and inverse dynamics under one conditional diffusion framework (Zhang et al., 29 Sep 2025).