KeyDiff3D: Unsupervised Monocular 3D Keypoints

• KeyDiff3D is a framework for unsupervised monocular 3D keypoints estimation that leverages a pretrained multi‐view diffusion model to convert implicit 3D priors into explicit representations.
• It transforms multi‐view diffusion features into unified 3D feature volumes and keypoint activation maps using differentiable unprojection and soft attention aggregation.
• The method enables end‐to‐end 3D object manipulation by reconstructing meshes and binding them to a skeleton derived from learned keypoint structures.

Searching arXiv for the specified paper and directly related works. KeyDiff3D is a framework for unsupervised monocular 3D keypoints estimation that predicts 3D keypoints from a single image while using only a collection of single-view images for training. It is designed for the stronger setting in which there are no 3D labels, no 2D keypoint labels, no real multi-view supervision, and no camera calibration on real data. The method leverages geometric priors embedded in a pretrained multi-view diffusion model, converts the model’s implicit 3D priors into explicit 3D feature volumes and keypoints, and further introduces a pipeline for manipulating 3D objects generated by the diffusion model from a single image (Jeon et al., 16 Jul 2025).

1. Problem setting and scope

KeyDiff3D addresses unsupervised monocular 3D keypoint discovery. Given a single RGB image $I$ of an object, the objective is to predict a set of $N$ 3D keypoints

$$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$

together with a structural graph over these keypoints represented by an adjacency matrix $\mathcal{A}\in\mathbb{R}^{N\times N}$ with learnable edge weights $e_{i,j}$ (Jeon et al., 16 Jul 2025).

The formulation excludes several forms of supervision that are standard in prior work: ground-truth 3D joint locations, 2D keypoint annotations, real multi-view data, and calibrated cameras. Training data therefore consists only of independent single-view images of an object category. The paper frames this as difficult because monocular input is subject to depth ambiguity, occlusion, unknown camera parameters, and the absence of labels. Previous approaches mitigate these issues with calibrated multi-view images, multi-view reconstruction losses, or category-specific kinematic priors, but those requirements are expensive and not easily scalable to arbitrary categories (Jeon et al., 16 Jul 2025).

Within this landscape, KeyDiff3D targets a setting in which both training and inference are monocular, manual labels are unavailable, and real multi-view capture is not used. The enabling component is a pretrained multi-view diffusion model trained elsewhere, which is treated as a source of strong implicit 3D knowledge. This suggests a shift in supervision strategy: rather than collecting geometric annotations or real synchronized views, the method derives supervisory structure from a generative model’s learned multi-view consistency (Jeon et al., 16 Jul 2025).

2. Multi-view diffusion prior

The diffusion backbone is SV3D-p, described as a latent diffusion model that takes a single conditioning image $I$ and a set of target camera projection matrices $\{\mathbf{P}_k\}_{k=1}^K$, and generates a set of geometrically consistent images $\{I^{(k)}\}_{k=1}^K$ corresponding to those views (Jeon et al., 16 Jul 2025). In latent space it operates on

$z_t\in\mathbb{R}^{C\times K\times H\times W},$

and at each diffusion timestep a U-Net predicts noise residuals jointly for all views, conditioned on the input image and camera parameters $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$. The joint modeling of views induces multi-view consistency, so that pixels across views for the same 3D point tend to be coherent (Jeon et al., 16 Jul 2025).

For each real training image, KeyDiff3D first extracts an object mask via Grounded-SAM and crops or masks the background so that diffusion focuses on the object. It then specifies target cameras $N$0. Although SV3D-p uses 21 candidate views, KeyDiff3D uses $N$1: the original view and 3 novel views. Diffusion sampling is run from $N$2 down to a chosen timestep $N$3, with $N$4 and $N$5. At timestep $N$6, the method caches intermediate decoder features $N$7 for each U-Net layer $N$8, where

$N$9

Sampling is then completed to generate the multi-view images $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$0 used as supervision (Jeon et al., 16 Jul 2025).

The diffusion model therefore serves two distinct roles. It is a multi-view image generator that provides synthetic views with known virtual cameras, and it is a geometry-aware feature extractor whose intermediate activations encode semantic and geometric regularities. KeyDiff3D uses both outputs jointly: the images supervise reconstruction, while the cached U-Net features are lifted into explicit 3D representations (Jeon et al., 16 Jul 2025).

3. Architecture and explicit 3D representation

The architectural core begins with diffusion feature aggregation. Multi-layer diffusion features $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$1 are upsampled to a common spatial resolution $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$2, passed through bottlenecks $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$3 to a unified channel dimension $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$4, and combined with learnable scalar weights $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$5: $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$6 The paper notes that this is analogous to HyperFeatures, but here the downstream objective is to obtain features that are stable and structurally meaningful across views (Jeon et al., 16 Jul 2025).

A shallow keypoint head $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$7 maps the aggregated tensor to keypoint-specific 2D maps per view: $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$8 For each view $$\mathbf{S}=\{\mathbf{s}_n\}_{n=1}^N,\qquad \mathbf{s}_n\in\mathbb{R}^3,$$9, $\mathcal{A}\in\mathbb{R}^{N\times N}$0 can be interpreted as a stack of $\mathcal{A}\in\mathbb{R}^{N\times N}$1 keypoint activation maps (Jeon et al., 16 Jul 2025).

The central geometric step is differentiable unprojection into a 3D volume. A uniform $\mathcal{A}\in\mathbb{R}^{N\times N}$2 grid is defined in a canonical coordinate system, with voxel centers $\mathcal{A}\in\mathbb{R}^{N\times N}$3 and $\mathcal{A}\in\mathbb{R}^{N\times N}$4. For each voxel center and each view, projection is performed with

$\mathcal{A}\in\mathbb{R}^{N\times N}$5

$\mathcal{A}\in\mathbb{R}^{N\times N}$6

followed by normalization to image coordinates: $\mathcal{A}\in\mathbb{R}^{N\times N}$7 Bilinear sampling retrieves view-specific features at the projected location,

$\mathcal{A}\in\mathbb{R}^{N\times N}$8

and these are aggregated across views with a softmax-based attention over the view dimension: $\mathcal{A}\in\mathbb{R}^{N\times N}$9

$e_{i,j}$0

Collecting all voxels yields

$e_{i,j}$1

This volume is processed by a 3D convolutional network $e_{i,j}$2 to produce per-keypoint 3D heatmaps $e_{i,j}$3, which are converted to continuous coordinates with integral regression: $e_{i,j}$4

$e_{i,j}$5

The output is an ordered set of 3D keypoints in the same canonical frame as the diffusion model’s latent geometry (Jeon et al., 16 Jul 2025).

4. Self-supervised training objective

Training is unsupervised in the sense that the supervisory target is not human annotation but the diffusion-generated views themselves. Once the 3D keypoints $e_{i,j}$6 are predicted, they are projected to each virtual view: $e_{i,j}$7 and normalized to obtain $e_{i,j}$8 (Jeon et al., 16 Jul 2025).

To impose structure, the method constructs a differentiable edge map $e_{i,j}$9 in each view. The learnable adjacency matrix $I$0 supplies weights $I$1 for pairs of keypoints. For each pair $I$2, a soft line $I$3 is drawn between the projected keypoints $I$4 and $I$5, with Gaussian falloff, and the lines are aggregated as

$I$6

The resulting map serves as a per-pixel representation of the structure induced by the predicted 3D keypoints (Jeon et al., 16 Jul 2025).

Reconstruction provides the principal loss. The original image is randomly affinely augmented, a low-level appearance feature is extracted, and a reconstruction network receives the appearance feature together with $I$7 to produce

$I$8

The target is the diffusion-generated image $I$9. Two losses are applied per view: $\{\mathbf{P}_k\}_{k=1}^K$0 where $\{\mathbf{P}_k\}_{k=1}^K$1 is a VGG feature extractor, and

$\{\mathbf{P}_k\}_{k=1}^K$2

where $\{\mathbf{P}_k\}_{k=1}^K$3 is a binary foreground mask extracted from the diffusion-generated image. The total loss is

$\{\mathbf{P}_k\}_{k=1}^K$4

with $\{\mathbf{P}_k\}_{k=1}^K$5 and $\{\mathbf{P}_k\}_{k=1}^K$6 (Jeon et al., 16 Jul 2025).

No explicit multi-view consistency loss is written directly over keypoints. Instead, consistency arises implicitly from the use of shared 3D keypoints, shared volumetric representation, known camera projections, and multi-view reconstruction. The paper states that no hand-engineered equivariance or separation losses are required (Jeon et al., 16 Jul 2025).

5. 3D object manipulation pipeline

Beyond keypoint estimation, KeyDiff3D introduces a pipeline for manipulating 3D objects generated by the diffusion model from a single image. The first stage reconstructs a 3D object from diffusion-generated multi-view images and their camera parameters using Gaussian Frosting, which is described as being built on 3D Gaussian Splatting. Gaussian Frosting reconstructs both a set of 3D Gaussians approximating the radiance field and an associated mesh with vertices $\{\mathbf{P}_k\}_{k=1}^K$7 (Jeon et al., 16 Jul 2025).

The discovered keypoints $\{\mathbf{P}_k\}_{k=1}^K$8 and dense learnable adjacency graph $\{\mathbf{P}_k\}_{k=1}^K$9 are then converted into a tree-like skeleton. A Minimum Spanning Tree is constructed over the keypoints, using an edge weight that combines predicted adjacency weights $\{I^{(k)}\}_{k=1}^K$0 and Euclidean distances $\{I^{(k)}\}_{k=1}^K$1. This yields a sparse connected structure with $\{I^{(k)}\}_{k=1}^K$2 edges $\{I^{(k)}\}_{k=1}^K$3, suitable for articulation (Jeon et al., 16 Jul 2025).

Skinning binds the mesh to this skeleton. For mesh vertex $\{I^{(k)}\}_{k=1}^K$4 and skeleton edge $\{I^{(k)}\}_{k=1}^K$5, a Gaussian-based skinning weight is defined by

$\{I^{(k)}\}_{k=1}^K$6

where $\{I^{(k)}\}_{k=1}^K$7 is the minimum distance from the vertex to the edge in 3D, and $\{I^{(k)}\}_{k=1}^K$8 control falloff. The normalized weight is

$\{I^{(k)}\}_{k=1}^K$9

Once mesh vertices, skeleton edges, and skinning weights are available, standard articulation operations can be applied by transforming bones and blending the resulting vertex motions. Because the keypoints are learned in the same coordinate frame as the diffusion model and the multi-view reconstructions, no additional registration is needed (Jeon et al., 16 Jul 2025).

The paper presents this as enabling moving limbs, bending bodies, and simple reposing and animation of generated objects from a single input image. A plausible implication is that the discovered 3D structure is not only a latent evaluation target but also a manipulable intermediate representation for downstream graphics-oriented tasks (Jeon et al., 16 Jul 2025).

6. Experimental evaluation

The experimental study covers Human3.6M, Stanford Dogs, AP-10K, DAVIS, and Google Scanned Objects. On Human3.6M, the multi-view and camera parameters are ignored in the main setting and each frame is treated as an independent single-view image; training uses subjects 1, 5, 6, 7, 8 and testing uses 9, 11. Stanford Dogs contributes 3000 training images with full-body visible instances, with training on 92 breeds and testing on 28 unseen breeds. AP-10K is used only for testing cross-species generalization, DAVIS for qualitative in-the-wild human results, and Google Scanned Objects for out-of-domain generalization to toys, characters, and household objects (Jeon et al., 16 Jul 2025).

Evaluation follows the standard probing protocol for unsupervised keypoint discovery. KeyDiff3D is trained without labels, frozen, and then a small regressor is trained from the discovered 3D keypoints to ground-truth joint positions on the training split. The regressor is either a linear regressor with no bias or an MLP with 2 hidden layers of size 50 each. Metrics are Mean Per Joint Position Error, N-MPJPE, and P-MPJPE (Jeon et al., 16 Jul 2025).

Method or setting	MPJPE	P-MPJPE
KeyDiff3D, 18 keypoints, 2-layer MLP regressor	121.34 mm	85.26 mm
KeypointNet	158.7 mm	112.9 mm
Honari et al.	125.73 mm	89.05 mm

On Human3.6M, KeyDiff3D with 18 keypoints and a 2-layer MLP regressor achieves MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$0 mm, N-MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$1 mm, and P-MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$2 mm. The paper reports that this outperforms unsupervised single-view baselines and is competitive with some multi-view-based or human-specific methods, despite using no 2D or 3D labels and no real multi-view data. On a simplified subset of 6 actions, KeyDiff3D reaches P-MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$3 mm (Jeon et al., 16 Jul 2025).

Qualitatively, the paper states that on Human3.6M the method localizes joints accurately under varied poses and strong occlusions, such as a hand behind the torso. Trained solely on indoor Human3.6M, it still predicts reasonable skeletons on DAVIS. Trained only on human images, it produces consistent keypoints on robots, dolls, and game characters in Google Scanned Objects. On Stanford Dogs and AP-10K species including zebra, rhino, and giraffe, the predicted keypoints fall on meaningful anatomical locations such as the head, limbs, and tail, without animal-specific priors or labels (Jeon et al., 16 Jul 2025).

7. Ablations, limitations, and relation to prior work

Ablation studies isolate the contribution of diffusion features. When the diffusion feature extractor is replaced with a standard ResNet applied to each generated view, while keeping diffusion-generated images and the same unprojection-and-aggregation pipeline, the Human3.6M results worsen. Without diffusion features, the paper reports MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$4 with a linear regressor and $z_t\in\mathbb{R}^{C\times K\times H\times W},$5 with an MLP, and P-MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$6 with a linear regressor and $z_t\in\mathbb{R}^{C\times K\times H\times W},$7 with an MLP. With diffusion features, the corresponding values are MPJPE $z_t\in\mathbb{R}^{C\times K\times H\times W},$8 and $z_t\in\mathbb{R}^{C\times K\times H\times W},$9, and P-MPJPE $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$0 and $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$1. The paper interprets this as evidence that having multi-view images alone is insufficient, and that the geometry-aware diffusion representations are crucial (Jeon et al., 16 Jul 2025).

A second ablation varies the number of diffusion views $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$2. For $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$3, MPJPE is $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$4 and P-MPJPE is $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$5. For $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$6, the numbers are $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$7 and $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$8. For $$\mathbf{P}_k=\mathbf{K}[\mathbf{R}_k\mid \mathbf{t}_k]$$9, they are $N$00 and $N$01. For the default $N$02, they are $N$03 and $N$04. For $N$05, they are $N$06 and $N$07. The paper observes that one view severely underperforms, that adding even a single extra virtual view drastically improves performance, and that gains saturate around 3–5 views, motivating the choice $N$08 as a trade-off (Jeon et al., 16 Jul 2025).

The reported failure modes and limitations are tied to the dependence on the pretrained multi-view diffusion model. If that model fails to produce geometrically consistent views, as in extreme articulations or rare object shapes, keypoint accuracy degrades. Inference is relatively heavy because the method runs part of the diffusion denoising process to timestep $N$09 for each image and performs 3D convolution over a $N$10 voxel grid. The object-centric masking also ignores scene context, which the paper notes could be useful for some tasks (Jeon et al., 16 Jul 2025).

In relation to prior work, KeyDiff3D is positioned at the intersection of unsupervised keypoint discovery, monocular 3D pose estimation, and the use of generative models as geometry priors. The paper contrasts it with methods that rely on real multi-view or video supervision, 2D keypoints, unpaired 3D priors, temporal consistency, or category-specific skeleton priors, and with approaches such as KeypointNet, BKinD-3D, Honari et al., Thewlis, Jakab, Lorenz, StableKeypoints, and mask-based monocular human pose baselines (Jeon et al., 16 Jul 2025). The specific novelty claims are unsupervised monocular 3D keypoints from single images without real multi-view data, 2D labels, 3D labels, or category-specific skeletons; an explicit 3D feature volume constructed from multi-view diffusion features through differentiable unprojection and attention aggregation; structural training via multi-view reconstruction from diffusion-generated views; and an end-to-end pipeline from 2D image to 3D keypoints, 3D mesh, and animatable 3D object (Jeon et al., 16 Jul 2025).