OnePoseViaGen: Novel 6D Pose Estimation

• OnePoseViaGen is a dual-method approach that estimates object pose using either diffusion denoising with RGB images or single-view 3D mesh alignment with RGB-D inputs.
• It combines innovative techniques such as two-side novel view generation, metric scale recovery, and coarse-to-fine refinement to bypass the need for multi-view training or CAD models.
• The framework employs text-guided generative domain randomization to robustly refine pose estimates under occlusion, lighting changes, and large viewpoint shifts.

OnePoseViaGen is a name used for two distinct single-reference object-pose estimation pipelines in recent arXiv literature. One formulation, introduced in "Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching," determines object pose from a single reference RGB image and a query RGB image by combining diffusion-based novel-view generation with two-sided matching in diffusion-denoising space, without 3D object models or multi-view training (Sun et al., 2024). A later formulation, presented in "One View, Many Worlds: Single-Image to 3D Object Meets Generative Domain Randomization for One-Shot 6D Pose Estimation," starts from a single RGB-D anchor image, constructs a normalized mesh, recovers metric scale and 6D pose through coarse-to-fine alignment, and uses text-guided generative domain randomization to fine-tune a pose estimator (Geng et al., 9 Sep 2025). The shared label therefore spans substantially different assumptions, intermediate representations, and optimization strategies.

1. Problem definitions and system-level organization

The earlier OnePoseViaGen formulation takes as input a single reference RGB image $I_r$ of an unseen rigid object, whose absolute pose $(\phi_r,\theta_r)$ in the object frame is known, and a query RGB image $I_q$ of the same object under an unknown pose $(\phi_q,\theta_q)$. Its stated goal is to recover $\phi_q$ and $\theta_q$ from $I_q$ and $I_r$ without any 3D model or multi-view training. The later formulation instead takes a single RGB-D anchor image $I_A$, outputs a textured 3D mesh $O_M$ in metric scale and its 6D pose $(\phi_r,\theta_r)$0, and then uses synthetic diversification to support one-shot 6D pose estimation (Sun et al., 2024, Geng et al., 9 Sep 2025).

Work	Input / output	Core mechanism
(Sun et al., 2024)	$(\phi_r,\theta_r)$1 $(\phi_r,\theta_r)$2 recover $(\phi_r,\theta_r)$3	Zero-1-to-3 generation + two-side matching
(Geng et al., 9 Sep 2025)	$(\phi_r,\theta_r)$4 $(\phi_r,\theta_r)$5 $(\phi_r,\theta_r)$6	Hi3DGen + coarse-to-fine alignment + Trellis diversification

This distinction is central for interpreting reported results. The RGB-only system is framed around generalizable pose estimation with no object-specific training, whereas the RGB-D system is framed around single-view 3D generation, metric scale recovery, render-and-compare refinement, and synthetic-data-driven fine-tuning. A plausible implication is that the name OnePoseViaGen is better understood as an overloaded label than as a single stable method family.

2. Two-side generating and matching in diffusion-denoising space

In the 2024 formulation, the pipeline begins by fixing a set of $(\phi_r,\theta_r)$7 “intermediate” camera poses $(\phi_r,\theta_r)$8 sampled over the object’s upper hemisphere. A pre-trained latent diffusion model $(\phi_r,\theta_r)$9, specifically Zero-1-to-3, is then used to generate for each intermediate pose two small-baseline views: one from the reference image and one from a candidate query pose. These are written as

$I_q$0

with $I_q$1, $I_q$2, and analogous definitions for the query-conditioned offsets (Sun et al., 2024).

The diffusion model is described through the standard forward process

$I_q$3

and a reverse denoiser

$I_q$4

which predicts the injected noise. In OnePoseViaGen, the pre-trained Zero-1-to-3 denoiser

$I_q$5

is iterated from $I_q$6 down to $I_q$7 to generate clean novel views. The paper attributes generalization to unseen objects to the fact that Zero-1-to-3 was trained on thousands of objects at many relative viewpoints and is used here without further fine-tuning.

The central methodological departure is the replacement of direct pixel matching with matching of denoiser residuals. For a fixed diffusion time $I_q$8 and Monte Carlo samples $I_q$9, the method forms

$(\phi_q,\theta_q)$0

feeds this noisy version of the reference-generated view into the denoiser conditioned on the query image and the candidate relative viewpoint, and measures the residual

$(\phi_q,\theta_q)$1

Summing over intermediate viewpoints and averaging over noise samples yields the matching score

$(\phi_q,\theta_q)$2

The stated intuition is that if the assumed $(\phi_q,\theta_q)$3 is correct, then the denoiser conditioned on the query image and the corresponding relative viewpoint will exactly remove the injected noise from $(\phi_q,\theta_q)$4, driving $(\phi_q,\theta_q)$5 toward zero.

3. Pose search, training regime, and reported accuracy of the RGB-only pipeline

Pose recovery in the 2024 system is organized as a coarse-to-fine search followed by continuous refinement. Elevation is first initialized with an off-the-shelf single-image elevation network, exemplified by One-2-3-45. Azimuth is then searched coarsely over $(\phi_q,\theta_q)$6, followed by local searches at $(\phi_q,\theta_q)$7 and then $(\phi_q,\theta_q)$8. Elevation is perturbed around its initial guess in coarse intervals to further minimize the matching score. The final refinement step treats $(\phi_q,\theta_q)$9 as continuous and performs stochastic gradient descent on $\phi_q$0: $\phi_q$1 This optimization is coupled to a training regime with no object-specific training: the method leverages a pre-trained Zero-1-to-3 diffusion model and a pre-trained elevation predictor from One-2-3-45, with no fine-tuning on ShapeNet, CO3D, or any test-object CAD models (Sun et al., 2024).

Evaluation is reported on GSO, a synthetic benchmark with 23 test objects, and NAVI, a real benchmark with 27 test objects after filtering symmetric cases. The metric is rotation-error accuracy under thresholds $\phi_q$2 and $\phi_q$3. Using $\phi_q$4, $\phi_q$5, and $\phi_q$6, the GSO test set under full-view differences yields $\phi_q$7 of query poses within $\phi_q$8 and $\phi_q$9 within $\theta_q$0, compared with the best prior, E2VG with $\theta_q$1, at $\theta_q$2 and $\theta_q$3. On NAVI, OnePoseViaGen reports $\theta_q$4 at $\theta_q$5 and $\theta_q$6 at $\theta_q$7, compared with E2VG at $\theta_q$8 and $\theta_q$9. Under large viewpoint shifts $I_q$0 on GSO, the reported numbers are $I_q$1 and $I_q$2 for OnePoseViaGen versus $I_q$3 and $I_q$4 for E2VG.

The method is explicitly contrasted with classical matching pipelines such as SIFT+PnP and LoFTR+RANSAC, with recent single-image or diffusion-based methods such as IDPose and RelPose++, and with one-side generative pipelines such as E2VG and IDPose. The paper’s claim is that the two-side paradigm addresses the view-gap problem by generating intermediate small-baseline views from both images and matching them in diffusion-denoising space rather than by naively matching a single generated side.

4. Single-view 3D generation, scale recovery, and coarse-to-fine alignment

The 2025 formulation redefines OnePoseViaGen around a geometric reconstruction-and-alignment stack. It builds on Hi3DGen to predict a normalized mesh $I_q$5 in a unit bounding sphere, then estimates a fully metric 6D pose by combining multi-view feature matching, scale recovery, and render-and-compare refinement. The coarse stage uniformly renders the normalized mesh under $I_q$6 spherical viewpoints $I_q$7, extracts SuperPoint features on each render, matches them to features in the RGB-D anchor image $I_q$8 via SuperGlue, and lifts the corresponding points to 3D as

$I_q$9

PnP on $I_r$0 provides an initial rotation $I_r$1 and translation $I_r$2 (Geng et al., 9 Sep 2025).

Because the mesh is scale-normalized, a global scale factor $I_r$3 is recovered by minimizing

$I_r$4

with closed-form solution

$I_r$5

The coarse estimate is then

$I_r$6

The fine stage adapts a small neural pose-refinement network from FoundationPose. Starting from $I_r$7, the method renders $I_r$8 under $I_r$9 and compares it against the observed depth or RGB in $I_A$0. The network predicts incremental updates $I_A$1, applied as

$I_A$2

After each update, scale is re-estimated through the same $I_A$3, and the iteration continues until convergence, yielding

$I_A$4

The paper also states a generic matching loss

$I_A$5

and a render-and-compare loss

$I_A$6

5. Text-guided generative domain randomization and quantitative evaluation

The second OnePoseViaGen adds a text-guided generative domain randomization module to bridge the “generated→real” domain gap. The normalized mesh $I_A$7, together with a short text prompt, is fed into the Trellis text-to-3D model, which produces multiple texture-diverse variants $I_A$8 while preserving the original geometry. These variants are rendered under randomized backgrounds, lighting, camera viewpoints, and random occluders to construct a large synthetic training set, and the render-and-compare refinement network is fine-tuned on this data (Geng et al., 9 Sep 2025).

The reported data-generation procedure is explicit. For each novel object, the pipeline generates 100 text-diversified meshes via Trellis in approximately 10 min on an A800 GPU; samples 3 variants per scene plus the original for 4 targets; adds 10 random BOP occluders and random CCTextures background; samples 100 camera poses around the object with random lighting and occlusion; and renders RGB-D together with segmentation masks to obtain approximately 40k training images. The pose refinement network is FoundationPose-based, with a ResNet-style feature extractor and cross-attention, and predicts $I_A$9 in quaternion or axis-angle together with $O_M$0. The reported loss is

$O_M$1

augmented by $O_M$2. Optimization uses Adam with learning rate $O_M$3, batch size $O_M$4, and 50 K iterations, while fine-tuning the LoRA module takes approximately 50 min on one L20 GPU.

Benchmark results are reported on YCBInEOAT, Toyota-Light, and LINEMOD-O. On YCBInEOAT, using ADD-S / ADD, the paper reports $O_M$5, compared with Any6D at $O_M$6. On Toyota-Light, with BOP metrics AR, MSSD, MSPD, and VSD, the reported values are AR $O_M$7, MSSD $O_M$8, MSPD $O_M$9, and VSD $(\phi_r,\theta_r)$00, compared with Any6D at AR $(\phi_r,\theta_r)$01, MSSD $(\phi_r,\theta_r)$02, MSPD $(\phi_r,\theta_r)$03, and VSD $(\phi_r,\theta_r)$04. On LINEMOD-O, the paper reports AR $(\phi_r,\theta_r)$05, MSSD $(\phi_r,\theta_r)$06, MSPD $(\phi_r,\theta_r)$07, and VSD $(\phi_r,\theta_r)$08, compared with GigaPose at AR $(\phi_r,\theta_r)$09, MSSD $(\phi_r,\theta_r)$10, MSPD $(\phi_r,\theta_r)$11, and VSD $(\phi_r,\theta_r)$12. The ablation on Toyota-Light gives AR $(\phi_r,\theta_r)$13 for the full system with alignment, $(\phi_r,\theta_r)$14 without coarse alignment, and $(\phi_r,\theta_r)$15 without fine alignment; synthetic fine-tuning gives AR $(\phi_r,\theta_r)$16, compared with naive $(\phi_r,\theta_r)$17 and none $(\phi_r,\theta_r)$18. Qualitatively, the paper states that generated single-view meshes closely match ground-truth CAD, that coarse match plus refinement produces sub-centimeter alignment, and that render-compare selection yields robust 6D pose under heavy occlusion and lighting changes. For real-robot dexterous grasping with XHAND1, the reported pick-and-place success is $(\phi_r,\theta_r)$19, versus SRT3D at $(\phi_r,\theta_r)$20 and DeepAC at $(\phi_r,\theta_r)$21.

6. Relation to prior work, recurring failure modes, and interpretive boundaries

The 2024 OnePoseViaGen is positioned against three strands of prior work: feature matching, regression, and one-side diffusion-based generation. Its stated critique is that feature matching methods such as SIFT and LoFTR fail when the baseline exceeds $(\phi_r,\theta_r)$22; regression methods such as RelPose and RPR are category-limited or require 3D CADs; and diffusion-based E2VG or IDPose generate only one side and match naively, which degrades under large baselines. Its contribution is defined as a “two-side” paradigm that remains model-free and single-reference while using a score-distillation-like denoiser residual to reduce the view-gap problem (Sun et al., 2024).

The 2025 OnePoseViaGen is positioned differently. It assumes that single-view 3D generation can provide a normalized mesh, and that the residual mismatch between generated geometry and real observations can be mitigated through metric scale recovery, render-and-compare refinement, and synthetic fine-tuning from text-guided texture diversification. This suggests an overview between single-image-to-3D object reconstruction and one-shot 6D pose estimation rather than a purely zero-shot matching pipeline (Geng et al., 9 Sep 2025).

The limitations are likewise different. The first method emphasizes robustness under significant viewpoint changes, including $(\phi_r,\theta_r)$23, but its formulation is tied to recovering angular pose from RGB images with a known reference pose. The second paper explicitly lists deformable or highly articulated objects as violating the rigid alignment assumption, and states that severe motion blur, extreme symmetry, or textureless geometry can still fail. Its future directions are to incorporate test-time mesh refinement or continuous geometry adaptation, extend to non-rigid or articulated object pose, and combine with learning-based symmetry priors to disambiguate symmetric views.

Taken together, the literature under the name OnePoseViaGen covers two non-equivalent research programs. One uses a large pretrained diffusion prior to hallucinate controlled novel views and performs search in diffusion-denoising space without object-specific training. The other uses high-fidelity single-view 3D generation, explicit metric alignment, and generative domain randomization to support reliable one-shot 6D pose estimation. The shared name is therefore nominally continuous but technically discontinuous, and careful citation by arXiv identifier is necessary when comparing methods or reproducing results.