Exploring 6D Object Pose Estimation with Deformation

Abstract: We present DeSOPE, a large-scale dataset for 6DoF deformed objects. Most 6D object pose methods assume rigid or articulated objects, an assumption that fails in practice as objects deviate from their canonical shapes due to wear, impact, or deformation. To model this, we introduce the DeSOPE dataset, which features high-fidelity 3D scans of 26 common object categories, each captured in one canonical state and three deformed configurations, with accurate 3D registration to the canonical mesh. Additionally, it features an RGB-D dataset with 133K frames across diverse scenarios and 665K pose annotations produced via a semi-automatic pipeline. We begin by annotating 2D masks for each instance, then compute initial poses using an object pose method, refine them through an object-level SLAM system, and finally perform manual verification to produce the final annotations. We evaluate several object pose methods and find that performance drops sharply with increasing deformation, suggesting that robust handling of such deformations is critical for practical applications. The project page and dataset are available at https://desope-6d.github.io/}{https://desope-6d.github.io/.

• The paper introduces a novel dataset, DeSOPE, capturing 26 object classes in canonical and three deformation states for 6D pose estimation.
• It details a multi-stage pipeline including high-fidelity scanning, dense 3D registration, and extensive RGB-D annotation.
• Experimental results reveal a significant performance drop in pose estimation even under mild deformation and occlusion.

Exploring 6D Object Pose Estimation Under Deformation: The DeSOPE Dataset

Introduction

6D object pose estimation is a foundational problem in robotics, AR/VR, and computer vision, underpinning tasks such as manipulation, navigation, and scene understanding. The standard assumption in most pose estimation research and benchmarks is object rigidity—methods and datasets expect that seen instances match a known canonical mesh or CAD model. However, real-world scenarios routinely violate this assumption as everyday objects such as bottles, boxes, and cans undergo non-articulated deformations due to wear, impact, or handling. Current benchmarks fail to test or characterize these realistic but challenging conditions.

The paper "Exploring 6D Object Pose Estimation with Deformation" (2604.06720) introduces DeSOPE, the first large-scale dataset designed for 6D object pose estimation in the presence of non-articulated, non-trivial deformation. DeSOPE provides high-fidelity 3D scans for 26 object categories, each captured in one canonical and three distinct deformed states, as well as aligned and annotated RGB-D image sequences across varied real-world environments.

Figure 1: Comparative illustration of canonical and deformed objects from DeSOPE, highlighting the dataset's focus on deformation states rather than solely rigid object assumptions.

Limitations of Existing Datasets and Motivation for DeSOPE

Existing datasets fall into two main paradigms: instance-level and category-level pose datasets. Instance-level datasets (e.g., LINEMOD, T-LESS, YCB-V) represent each instance with an individual 3D model, assuming perfect rigidity, whereas category-level datasets assign a single canonical mesh to all instances within a category. Neither approach captures the intra-instance geometry changes imposed by deformation, and neither provides aligned 3D scans for distinct deformation states.

Figure 1: Comparison of 6D object pose datasets, illustrating the unique contribution of DeSOPE in modeling multiple registered deformations per object.

DeSOPE closes this gap by:

• Scanning each of 26 object classes in their canonical form and three progressive deformation levels (mild, moderate, severe).
• Registering each deformed mesh to the canonical mesh via a dense flow-driven 3D alignment framework, providing tightly aligned ground truth for geometric evaluation.
• Densely annotating 133,000 RGB-D frames, each with up to five objects, yielding 665,000 pose annotations across both static and human-manipulated scenarios.

  Figure 2: Overview of the DeSOPE dataset generation framework, showing scanning, model alignment, video capture, and pose annotation pipeline.

Data Acquisition and Annotation Pipeline

The DeSOPE asset pipeline is comprised of the following stages:

• Object Scanning: A high-accuracy scanner is used to acquire detailed meshes of each object's canonical and three deformed states.
• 3D Registration: The deformed meshes are manually coarsely aligned, then refined using a state-of-the-art, flow-based multi-view registration algorithm (SCFlow2). Dense 2D-2D correspondences from six orthogonal projections are lifted to establish 3D-3D mappings, and outlier rejection plus similarity transformation (Umeyama) are used for precise registration.

Figure 3: Example of 3D model alignment—comparison of initial manual alignment versus refined registered results using dense multi-view flow, visualized with error maps.

• RGB-D Video Capture: Each scene is recorded in five different real-world indoor environments with both static and dynamic (human-manipulated) conditions. Diverse occlusion, viewpoint, and lighting variations are emphasized.
• Pose Annotation: 2D segmentation masks are generated with SAM2. Initial pose hypotheses are generated via FoundationPose, and further refined by object-level neural implicit optimization (Co-SLAM backbone), followed by manual verification, especially in challenging frames.

Figure 4: Example of captured images with annotated pose projections demonstrating dataset realism and scene complexity.

Quantitative Dataset Analysis

Comprehensive statistics confirm that DeSOPE provides:

• Broad coverage of object dimensions, geometries, and deformation types
• Uniform sampling of camera-object relative poses in SE(3)
• Fine-grained grading of deformation severity based on mean mesh pointwise displacement

Pose annotations are provided at scale, with high intra- and inter-scene variability as shown by coverage histograms.

Experimental Benchmarking

Three diverse state-of-the-art 6D pose estimation methods are evaluated without additional fine-tuning:

• SCFlow2: flow-based instance-level pose refiner
• FoundationPose: foundation-model object pose estimator (novel object inference)
• GenPose: generative diffusion-based category-level method

Results indicate that all methods perform robustly under rigid, canonical geometry. However, even mild deformation (Deformed 1) causes a significant drop in average recall—e.g., for SCFlow2, from 0.82 (canonical) to 0.23 (severely deformed) for static scenarios. The discrepancy is even more pronounced in human-manipulated sequences and under occlusion.

Figure 5: Visualization of SOTA method performance on canonical (top row) versus deformed (bottom row) objects in DeSOPE, highlighting pronounced accuracy degradation.

Analysis of Deformation and Occlusion Effects

Performance consistently decreases with increasing deformation severity across all methods and scenarios. Occlusion and human manipulation impose further substantial error, due to additional visual ambiguities and motion-induced depth noise. Category-level models like GenPose experience a smaller relative drop due to enhanced generalization, but none of the tested models are robust to non-articulated deformation.

Implications and Future Directions

The results establish that the rigid-object assumption is a critical, overlooked bottleneck for deployment of 6D pose estimation in practical environments. DeSOPE's aligned deformed meshes and large-scale annotated RGB-D data provide a platform for:

• Training and evaluation of deformation-aware geometric representations and pose estimators
• Development of hybrid models that disentangle pose and non-rigid shape, perhaps leveraging simulation or learned deformation priors
• Temporal modeling approaches, such as spatiotemporal neural implicit representations, that fuse information over time to resolve deformation ambiguity
• Extensions into differentiable physics and shape-from-interaction to recover deformation states jointly with 6D pose

The DeSOPE benchmark encourages the community to reconsider the brittle rigidity paradigm and advance toward models capable of robust performance in the presence of real-world deviations.

Conclusion

DeSOPE is a significant dataset resource that exposes and quantifies a core limitation of current pose estimation algorithms: their inability to handle nontrivial, real-world object deformations. Benchmarking indicates that pose accuracy degrades sharply with increasing deformation, regardless of method family. By enabling systematic study and method development for deformation-aware 6D object pose estimation, DeSOPE sets a new baseline for realism and practical relevance in the field.

---

Reference:

"Exploring 6D Object Pose Estimation with Deformation" (2604.06720)